Adaptive graphic user interfacing system

ABSTRACT

A system and method for an adaptive graphic user interface (AGUI) ( 501 ) that can identify the characteristics of one or more users and display candidates and dynamically select display content and generate an interface ( 502 - 506  . . . ) customized to one or more users and display candidates and dynamically arrange, modify, map and adapt an AGUI display to any object ( 350,730 ), device, clothing, equipment, furniture, appliance, room, building, vehicle, person, or other passive or active display surface, screen, space or other display candidate and embodiments of an AGUI apparatus ( 500 ) that can network, coordinate and/or operate with the AGUI platform ( 1302 ) to identify one or more users and display candidates and enabling one or more users to interface with a single AGUI display across multiple display candidates.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority under 35 U.S.C. 119(e) basedupon co-pending U.S. Provisional Application No. 62/575,424 filed onOct. 21, 2017. The entire disclosure of the prior application isincorporated herein by reference.

FIELD OF THE INVENTION

This invention relates to the field of computer display and userinterface technology. More particularly, the invention relates todevices and methods for adaptively displaying information on a varyingnumber of display surfaces while allowing a user to interact with them.

BACKGROUND

Existing electronic display devices include computer displays, phones,tablets, televisions, projectors and other similar devices. Manycomputer systems may also use multiple displays that are directlyconnected through a display cable or a wired or a wireless network. Theelectronic device may mirror the information it is displaying on eachattached display so that the same information is displayed on eachdisplay, may span its display across the multiple displays so that thegroup of displays operate as one large display, or may display differentinformation on each of the attached displays. Displays are typicallyrectangular in shape and are flat. Recently some large displays havebeen manufactured with a gentle curve to adapt to a viewer's point ofview but are still generally flat. Curved displays have also beenprovided for cell phones, watches and televisions. Projectors displaytheir output on generally flat surfaces using the resolution of theprojector. Keystone correction is frequently done to ensure that theimage displayed appears square even though the projection may not belocated directly in front of the display surface. Though these displaysare all fixed in size and/or shape there are other display technologiessuch as printed, painted, projected and flexible displays that are lessfixed in size, shape and other variables.

User interface control devices include keyboard, touchscreen and pointertechnology such a computer mouse that are basically x-y positional inputdevices. Some systems track the position of a device, pointer, or bodypart in 3-D space. Other wearable optical hand, finger and objectspatial positioning systems incorporate gesture and voice recognition aswell as touchscreen interfacing controls.

Display and user interface technology may be merged with a projectorbeing used to project a virtual graphic user interface on a surface suchas a desk, wall, or a body part such as a hand. By touching theprojected graphic user interface or by making gestures, the systemdetects the motion or placement of a pointer, which allows a user tointerface with the system.

A drawback of the existing technology is that the present day solutionsare inflexible and limited in their use. Display setups incorporatinguser interfaces are difficult to set up, configure, and interact with.There is a need for an improved graphical user interface BRIEF SUMMARY

According to some aspects and embodiments of the present technology,there is provided one or more system and/or one or more method for anadaptive graphic user interface (AGUI) that can identify thecharacteristics of one or more users and display candidates anddynamically select display content and generate an interface customizedto one or more users and display candidates and dynamically arrange,modify, map and adapt an AGUI display to any object, device, clothing,equipment, furniture, appliance, room, building, vehicle, person, orother passive or active display surface, screen, space or other displaycandidate and embodiments of an AGUI apparatus that can network,coordinate and/or operate with the AGUI platform to identify one or moreusers and display candidates and enabling one or more users to interfacewith a single AGUI display across multiple display candidates, enablingcustom AGUI displays for each display candidate, enabling one or moreusers to access their own custom AGUI display on one or more displaycandidates and enabling two or more AGUI apparatus to operate in concertto display a single or multiple AGUI displays on one or more displaycandidates.

According to some embodiments, a system is provided for an adaptivegraphic user interface (AGUI). The system can be configured to identifythe characteristics of one or more users and display candidates.Additionally or alternatively, the system can be configured todynamically select display content and/or generate an interface. Thesystem may be configured customize the interface to one or more usersand display candidates. The system may be further configured todynamically arrange, modify, map and/or adapt an AGUI display to anyobject. In some embodiments, the system is configured to dynamicallyarrange, modify, map and/or adapt the AGUI display to any one or anycombination of the following: a device, clothing, equipment, furniture,appliance, room, building, vehicle, person, other passive and/or activedisplay surface, screen, space, other display candidate, and embodimentsof an AGUI apparatus disclosed herein. Such embodiments of the AGUIapparatus can be further configured to network, coordinate and/oroperate with the AGUI platform to identify one or more users and displaycandidates. The aforesaid system(s) that are configured to dynamicallyarrange, modify, map and/or adapt the AGUI display and network,coordinate and/or operate with the AGUI platform enable one or moreusers to interface with a single AGUI display across multiple displaycandidates, enabling custom AGUI displays for each display candidate,enabling one or more users to access their own custom AGUI display onone or more display candidates and enabling two or more AGUI apparatusto operate in concert to display a single or multiple AGUI displays onone or more display candidates.

In some embodiments, the adaptive graphic user interface (AGUI) systemmay include an AGUI apparatus that may incorporate a computing device,group of devices and/or platform that comprises an embedded or networkedsensor for mapping an environment in proximity to the sensor to identifya user and a display candidate. A display output is configured fordisplaying content on a passive or active display candidate. A memorycontains an application coupled to a processor. The processor isconfigured to execute the application to generate display content. Theapplication evaluates the display candidate and the user position andselects the display candidate as a display surface or space. Theapplication identifies the user and the display candidate and selectsdisplay content and generates an interface customized for the user andthe display candidate. The application utilizes characteristics of thedisplay candidate and the user position to dynamically arrange, modify,map and adapt the display content and select the optimum interfacelayout to produce enhanced display content adapted to the displaycandidate and to the user. The display output displays an image of theenhanced display content on the display candidate. The display outputoperates in coordination with the sensor to enable the user to interfacewith the display candidate and control the display content in real-time.The application is continuously updating the user position and thedisplay candidate to detect perturbations in the environment. Inresponse to the perturbations, the application modifies the enhanceddisplay content to improve the image or detects a control action.

According to one aspect of the present technology, there is provided anadaptive graphic user interfacing system. This system can comprise asensor component configured or configurable to dynamically map anenvironment in proximity to the sensor component to detect a user and adisplay candidate. A display output component configured or configurableto identify a user and a display candidate and provide a display contenton or to the display candidate. At least one processing unit and/orplatform operably networked with and/or connected or connectable to thesensor component, the display component and at least one memory unit ornetworked remote data storage platform, the processing unit beingconfigured or configurable to determine a user's relational position tothe display candidate in real time, identify the display candidate andthe user in real time, and continuously adapt the display content basedon the user's relational position to the display candidate in real-timeas the user moves in relation to the display candidate and/or thedisplay candidate moves in relation to the user.

In some embodiments the display candidate is a passive surface and thedisplay output is a projector. In other embodiments, the displaycandidate is an active display surface such as an electroluminescent,backlit, organic, embedded or other self-illuminated display such as anLED, OLED, LCD or other active display system, screen, panel or materialin which the output is incorporated in the display surface.

In some embodiments the display candidate or candidates can be selectedfrom a group consisting of passive display systems and surfaces andactive display systems and surfaces.

In some embodiments, the display content can be selected from a groupconsisting of a graphical user interface, interactive media content suchas icons, images, video, text, interactive 3D objects, environments,buttons, and control affordances.

In some embodiments, the system can further comprise a microphone and/orspeaker component in operable communication with the processing unitand/or platform, and being configured or configurable to receive and/orprovide an audio signal.

In some embodiments one or more microphones and/or speakers in operablecommunication with the processing unit and/or platform may be configuredto provide an audio signal based on the spatial position of one or moreusers in relation to one or more microphones and/or speakers and maydynamically receive and/or provide customized audio content for eachuser and/or group of users.

A system and method for an adaptive graphic user interface (AGUI) thatcan dynamically adapt to any device, clothing, equipment, furniture,appliance, building, vehicle, person, or other object, display, screenor surface and embodiments of an AGUI apparatus that can support andcoordinate with an AGUI platform enabling one or more users to interfacewith a single AGUI interface across multiple devices, objects andsurfaces, enabling multiple users to access their own AGUI interface onmultiple devices, objects and surfaces and enabling multiple AGUIapparatus to operate in concert to display a single or multiple AGUIinterfaces on one or more stationary or moving devices, objects,vehicles or other surfaces.

In one approach of the present technology, an apparatus is providedwhich may comprise: a memory storing instructions; and one or moreprocessors, wherein said instructions, when processed by the one or moreprocessors, can cause: mapping, using a sensor, an environment inproximity to the apparatus to identify a user position and a displaycandidate; evaluating the display candidate and the user position;selecting the display candidate as a display surface; and utilizingcharacteristics of the display surface and the user position to generateor modify display content to produce enhanced display content. Theapparatus may include a display output and wherein said instructions,processed by the one or more processors, can further cause displaying animage of the enhanced display content on the display surface.

In some embodiments of the approach, a method of adapting a graphicaluser interface is provided which may comprise mapping an environment inproximity to the apparatus; identifying a user position and a displaycandidate from said mapping using a computer device; evaluating, usingsaid computer device, the display candidate and the user position;selecting the display candidate as a display surface; utilizingcharacteristics of the display surface and the user position to generateor modify display content, using said computer device, to produceenhanced display content; and displaying an image of the enhanceddisplay content on the display surface.

In some embodiments the approach, a computer-readable medium is providedincluding contents that are configured to cause a computing system toadapt a graphical user interface by performing a method comprising:mapping an environment in proximity to the apparatus; identifying a userposition and a display candidate from said mapping using a computerdevice; evaluating, using said computer device, the display candidateand the user position; selecting the display candidate as a displaysurface; utilizing characteristics of the display surface and the userposition to generate or modify display content, using said computerdevice, to produce enhanced display content; and displaying an image ofthe enhanced display content on the display surface.

In some aspects, a system and method for an adaptive graphic userinterface (AGUI) is provided that can dynamically adapt to any device,clothing, equipment, furniture, appliance, building, vehicle, person, orother object, display, screen or surface and embodiments of AGUI devicesthat can support and coordinate with an AGUI platform enabling one ormore users to interface with a single AGUI interface across multipledevices, objects and surfaces or multiple users to each access their ownindividual AGUI interface on different devices, objects and surfaces.

According to one aspect, there is provided a computing device comprisingone or more sensors for mapping and/or imaging an environment inproximity to the device to identify a user position and one or moredisplay candidates. A memory containing an application coupled to aprocessor executes an application to generate display content. Theapplication evaluates the display candidate and the user position andselects the display candidate as a display surface. The applicationutilizes characteristics of the display surface and the user position tomodify the display content to produce enhanced display content. Thedisplay output displays an image or images of the enhanced displaycontent on one or more of the display surfaces.

In some embodiments, the application continuously updates the positionof one or more users and one or more display candidates to detect aperturbation in the environment. In response to the perturbation, theapplication modifies the enhanced display content to improve the imageor the application detects a control action. In other embodiments, inresponse to the perturbation, the application detects a second displaycandidate and selects the second display candidate as the displaysurface. In other embodiments, the application detects two or moredisplay candidates and modifies the display content to multiple displaysurfaces.

In other embodiments, the application detects two or more displaycandidates and modifies the display content across two or more displaysurfaces. Modifying the display content may further comprise dividingthe display content between two or more display surfaces to produceenhanced display content for two or more display surfaces.

In other embodiments, the sensor identifies two or more displaycandidates and the application selects independent display content foreach display surface. In other embodiments, the sensor may assign onedisplay candidate as the master or control display and/or interfacingsurface and one or more other display candidates as a slave orsub-display and/or interfacing surface or surfaces.

In other embodiments, the sensor may assign one or more display surfacesas sub display surfaces and one or more users, objects or devices as themaster, input and/or control interface for one or all of the displaysurfaces.

In further embodiments the sensor may assign a different user, object,apparatus and/or interface to each display surface.

In other embodiments, the sensor identifies a second display candidateand the application selects the second display candidate as a seconddisplay surface. Modifying the display content further comprisesdividing the display content between the display surface and the seconddisplay surface to produce second enhanced display content for thesecond display surface. The display output displays a second image ofthe second enhanced display content on the second display surface or thedisplay output displays the same display content across two or moresurfaces or the display output displays different display content oneach display surface.

In other embodiments, the sensor may assign one display candidate as themaster or control display and/or interfacing surface and one or moreother display candidates as a slave or sub-display and/or interfacingsurface or surfaces.

In other embodiments, the sensor may assign one or more display surfacesas sub display surfaces and one or more users or devices as the master,input and/or control interface for all of the displays or the sensor mayassign a different device or interface to each display surface.

In some embodiments, the display surface is a passive surface and thedisplay output is a projector. In other embodiments, the display surfaceis an active display surface such as an electroluminescent, backlit,organic, embedded or other self-illuminated display such as an LED, LCD,OLED or other active display system, screen, panel, material in whichthe output is incorporated in the display surface.

In other embodiments, the computing device or plurality of computingsub-devices or display system may be mounted on or embedded in aceiling, wall, floor, window, desk or other furniture, appliance, objector surface.

In embodiments, the surface mounted computing device or plurality ofcomputing sub-devices or display system may be a multi-directionalsurface mapping, imaging, networking, display and interfacing system andmultimedia hub device further referred to collectively as “multimediahub device” capable of adapting a graphic user interface to one or moresurfaces and enabling one or more users to interface with projected oractively displayed content using touch, motion, gesture, voice commandand other interfacing methods. In further embodiments two or moremultimedia hub devices may be wired or wirelessly networked and mayoperate in concert as part of a group of devices in order to project orassign a graphic user interface and/or image, text, video or othermultimedia content to one or more surrounding surfaces.

In other embodiments the computing device, sub-devices or display systemmay be a handheld, portable, desktop or other mobile display and/orinterfacing system such as a phone, tablet, PC, laptop, touchpad, gameconsole, mouse, controller or other display and/or interfacing device,system, or surface.

In other embodiments the computing device, sub-devices or display systemmay be worn on the body such as on one or both wrists, in or on the earor worn on or in front of one or both eyes such as earphones,headphones, headsets, contact lenses, glasses, visors, helmets, cameras,projectors, screens and other wearable devices and systems worn on theface, head, chest or other body surface or attached to clothing,equipment or any combination of wearable device or sub-devices,

In other embodiments the computing device or sub-devices or displaysystem may be implanted into a users' brain, eyes, ears, skin or insidea user's body or embedded into or attached to a prosthetic deviceimplanted, attached, worn or otherwise assigned to a user.

In other embodiments of the invention the AGUI system may compriseorganic, flexible, malleable or other non-fixed form factor devices anddisplay systems such as fabric, clothing, shoes, hats, helmets and otherequipment, furniture, home, enterprise and vehicle upholstery and otherobjects, devices, display systems and surfaces which may naturally,mechanically or electronically change in shape, size or form andcomprises a plurality of light emitting elements that together comprisethe display surface.

In some embodiments, the computing device comprises a plurality ofcomputing sub-devices networked together. In other embodiments, thecomputing device or plurality of computing sub-devices comprises two ormore computing sub-devices mounted on or embedded in a ceiling, wall,floor, window, desk or other furniture, appliance, object or surface.

In other embodiments the computing device or sub-devices may be worn onthe body such as on one or both wrists, on the face, head, chest orother body surface or may be attached or embedded into clothing,equipment or any combination of wearable sub-devices.

In other embodiments the computing device or sub-devices may be ahandheld, portable, desktop or other mobile display and/or interfacingsystem such as a phone, tablet, PC, laptop, television, touchpad, gameconsole, mouse, controller or other display and/or interfacing device,system, or surface.

In further embodiments, the computing device comprises a plurality ofcomputing devices networked together and modifying the display contentfurther comprises allocating a first portion of the image to one of theplurality of computing devices and a second portion of the image toanother of the plurality of computing devices.

Some embodiments further comprise a network interface. The computerdevice detects a network device and determines that the network devicecan create a second display surface. Modifying the display contentfurther comprises dividing the display content between the displaysurface and the second display surface to produce second enhanceddisplay content for the second display surface. The computing devicetransmits the second enhanced display content to the network device fordisplay on the second display surface.

In further embodiments, the computing device is a head-up display suchas a pair of glasses, headset, visor, helmet or other wearable head-updisplay system which may incorporate any number of display methods. Insome embodiments, the head-up display system incorporates one or moreactive display surfaces such as an LED, OLED, LCD or other screen orscreens placed in front of one or both eyes or one or more picoprojectors for displaying an image on one or more lenses or otherdisplay surfaces in front of one or both eyes or reflecting an image offof one or more lenses or other display surfaces into one or both eyes orfor projecting display content directly into one or both eyes of a useror other head-up display method, screen or surface. The head-up displayscreen or surface may be transparent or have varying levels of opacityenabling the viewer to see their surrounding environment through thescreen, display surface or projected image and/or displayed content intotheir surrounding environment. The head-up display may also include oneor more cameras and sensors to capture a 2 dimensional, 3 dimensional or4 dimensional plenoptic image and/or video of their surroundingenvironment for display on one or more head-up display screens ordisplay surfaces in front of the eyes, on the surface of the eye orprojected into the eye and may assign an image or other graphic userinterface or display content into the captured image and/or video of theusers surrounding environment. The head-up display system may have oneor more methods for measuring depth of field and mapping objects,devices and surfaces surrounding the viewer such as light mapping and/oroptical imaging and other depth mapping, motion, orientation andlocation sensors enabling the head-up display system to identify itslocation and spatial position in relation to surrounding objects,devices and surfaces and assign displayed image, text, video or othermultimedia content to a depth of field and or assign interactivemultimedia content to a surface of the body of the viewer, a handheldobject or an object, device or surface surrounding the viewer or in thefield of view of the viewer.

In further embodiments, the computing device is incorporated into agarment that comprises a plurality of light emitting elements thattogether comprise the display surface.

In further embodiments of the organic and/or flexible display system theflexible material, surface and/or light emitting elements mayincorporate location, directional and/or orientational sensors thatindicate to the processor the spatial position, orientation anddirectional facing of the plurality of light emitting elements.

In further embodiments the organic and/or flexible display system, theflexible material, surface and/or light emitting elements mayincorporate light imaging, optical, haptic or other sensors for mapping,imaging or identifying surrounding objects and surfaces and/or to enablea user to interface with a graphic user interface and/or othermultimedia content displayed on the organic and/or flexible displaysurfaces. In other embodiments the computing device or sub-devices ordisplay system may be mounted on or embedded into one or more vehicles.

In other embodiments, the plurality of light emitting elementscomprises, LED, LCD, OLED or other light emitting elements or comprisesa directional sensor that indicated to the processor the facing of theplurality of light emitting elements.

In further embodiments, the computing device further comprises sportingequipment that comprise a second sensor in communication with theprocessor. The processor uses information received from the secondsensor to produce the enhanced display content.

In some embodiments, the device is mounted in a fixed location in theenvironment. In other embodiments, the device is mounted on a movableplatform in the environment.

According to another aspect, there is provided an imaging and displaydevice comprising an imaging system capturing a sequence of images in afield of view. A movement sensor outputs a plurality of movement dataindicative of the movement of the device within an environment. Alocation sensor outputs a plurality of location data indicative of alocation of the device. A processor module is coupled to the imagingsystem, the movement sensor, and the location sensor. The processormodule receives the sequence of images, the plurality of movement data,and the plurality of location data, and analyzes the sequence of imagesto determine if a control object in the field of view is in motion. Theprocessor module analyzes the plurality of movement data and determinesan orientation of the device. The processor module analyzes theplurality of movement data and the plurality of location data anddetermines the location of the device and determines if an event hasoccurred.

In further embodiments, the processor module further determines that theevent is a control gesture performed by the control object and theprocessor then executes a routine indicated by the control gesture. Inother embodiments, the processor module further determines that theevent is a perturbation of the location of the device or the orientationof the device and optimizing a display output of the device. If theprocessor module further determines that the event is a change in anenvironment of the device, the processor module reconfigures the deviceand/or the displayed content for the environment.

In some embodiments, the reconfiguration of the device compriseschoosing a new display surface.

According to the another aspect of the present technology; an imagingand display device is provided that may include: a memory storinginstructions; and one or more processors, wherein said instructions,when processed by the one or more processors, can cause:

-   -   receiving, from an imaging and display device, a sequence of        images, a plurality of movement data, and a plurality of        location data; analyzing the sequence of images to determine if        a control object in a field of view is in motion, analyzing the        plurality of movement data and determining an orientation of an        imaging and display device, analyzing the plurality of movement        data and the plurality of location data and determining the        location of the device, and determining if an event has        occurred.        In some embodiments, a method of imagining is provided which may        comprise:

receiving from an imaging and display device, a captured sequence ofimages in a field of view;

receiving from a movement sensor a plurality of movement data indicativeof the movement of the device within an environment;

receiving from a location sensor a plurality of location data indicativeof a location of the device;

analyzing the sequence of images to determine if a control object in thefield of view is in motion;

analyzing the plurality of movement data and determining an orientationof the device; and

analyzing the plurality of movement data and the plurality of locationdata and determining the location of the device, the processor moduledetermining if an event has occurred.

In some embodiments, a computer-readable medium is provided includingcontents that are configured to cause a computing system to analyzeimaging by performing a method comprising:

receiving from an imaging and display device, a captured sequence ofimages in a field of view;

receiving from a movement sensor a plurality of movement data indicativeof the movement of the device within an environment;

receiving from a location sensor a plurality of location data indicativeof a location of the device;

analyzing the sequence of images to determine if a control object in thefield of view is in motion;

analyzing the plurality of movement data and determining an orientationof the device; and

analyzing the plurality of movement data and the plurality of locationdata and determining the location of the device, the processor moduledetermining if an event has occurred.

According to another aspect, there is provided a computing devicedisplaying an image. The device comprises one or more sensors formapping a plurality of positions corresponding to a plurality ofviewers. Each of the plurality of positions comprises a viewer locationand a viewer orientation. A memory containing an application coupled toa processor, the processor executes the application to generate displaycontent on a plurality of surfaces. The application evaluates theplurality of positions and customizes the display content based on theplurality of surfaces. The application displays the display content onthe plurality of surfaces.

In some embodiments, a computer display device is provided that mayinclude: a memory storing instructions; and one or more processors,wherein said instructions, when processed by the one or more processors,can cause:

mapping, using a sensor, a plurality of positions corresponding to aplurality of viewers, each of the plurality of positions comprising aviewer location and a viewer orientation;

evaluating the plurality of positions and customizing the displaycontent based on the plurality of surfaces, and

displaying the display content on the plurality of surfaces.

In some embodiments method of adapting a graphical user interfacecomprises:

mapping, using a sensor a plurality of positions corresponding to aplurality of viewers, each of the plurality of positions comprising aviewer location and a viewer orientation;

evaluating the plurality of positions and customizing the displaycontent based on the plurality of surfaces, and displaying the displaycontent on the plurality of surfaces.

In some embodiments, a computer-readable medium is provided includingcontents that are configured to cause a computing system to adjust agraphical user interface by performing a method comprising:

mapping, using a sensor a plurality of positions corresponding to aplurality of viewers, each of the plurality of positions comprising aviewer location and a viewer orientation;

evaluating the plurality of positions and customizing the displaycontent based on the plurality of surfaces, and displaying the displaycontent on the plurality of surfaces.

In further embodiments, the environment is a vehicle. In otherembodiments the environment is the interior or exterior of a building.

In other embodiments, the device optimizes the display for each of oneor more viewers based on the role of each of the plurality of viewers inthe operation of the vehicle. In other embodiment, the device optimizesthe display for each of the plurality of viewers based on the positionof each of the one or more viewers inside of the vehicle.

In some embodiments, the position of each of the plurality of viewerscomprises a seating arrangement inside of the vehicle.

In other embodiments the device dynamically optimizes the display acrossone or more moving or stationary vehicles or other display surfaces.

In other embodiments the position of one or more viewers comprises theirviewing position outside of one or more vehicles or other displaysurfaces.

In other embodiments, the processor optimizes the display for each ofthe plurality of viewers based on a physical characteristic of each ofthe plurality of viewers. In some embodiment, the physicalcharacteristic comprises height.

In some embodiments, the system includes a plurality of light sensors.The processor is coupled to the plurality of light sensors and receivesambient light information from the plurality of light sensors inproximity to the plurality of users. The processor optimizes the displayfor each of the plurality of viewers based on the ambient lightinformation.

Further embodiments comprise an external light sensor. The processor iscoupled to the external light sensor and receives external lightinformation from the external light sensors. The processor utilizes theexternal light information when optimizing the display for each of theplurality of viewers.

In some embodiments, each of the plurality of viewers has a personalmobile device providing an identity of the plurality of viewers to theprocessor. The processor optimizes the display for each of the pluralityof viewers based on the identity of each of the plurality of viewers.

According to yet another aspect there is provided a computing deviceincorporated in a vehicle comprising a sensor for mapping an environmentin proximity to the vehicle to identify a location of the vehicle. Amemory containing an application coupled to a processor. The processorexecutes the application to generate display content. A display surfaceon an exterior surface of the vehicle. The application evaluates thelocation and modifies the display content to produce enhanced displaycontent based on the location. The enhanced display content is displayedon the display surface.

In some embodiments, a computer device is provided that may comprise: amemory storing instructions; and one or more processors, wherein saidinstructions, when processed by the one or more processors, can cause:mapping, using a sensor, an environment in proximity to a vehicle toidentify a location of the vehicle; evaluating the location of thevehicle, generating or modifying display content to produce enhanceddisplay content based on the location, and displaying enhanced displaycontent on display surface on an exterior surface of the vehicle.

In some embodiments, a method of adjusting a graphical user interface isprovided that may comprise mapping an environment in proximity to avehicle to identify a location of the vehicle; evaluating the locationof the vehicle, generating or modifying display content to produceenhanced display content based on the location, and displaying enhanceddisplay content on display surface on an exterior surface of thevehicle.

In some embodiments, a computer-readable medium is provided includingcontents that are configured to cause a computing system to adapt agraphical user interface by performing a method comprising: mapping anenvironment in proximity to a vehicle to identify a location of thevehicle; evaluating the location of the vehicle, generating or modifyingdisplay content to produce enhanced display content based on thelocation, and displaying enhanced display content on display surface onan exterior surface of the vehicle.

In some embodiments, the display surface comprises a vehicular bodywrap. The vehicular body wrap may comprise LED, OLED, LCD, glass, paintor other active display systems and elements or may comprise acombination of passive surfaces with projected content and activesurface displays. Displayed content may be coordinated across all activeand passive vehicle display surfaces by generating a depth and surfacemap of all passive and active vehicle display surfaces and then mapping,zoning and adapting content to individual panels and/or display surfacesand/or across multiple display surfaces or sections of the vehicle oracross the entire vehicle.

In further embodiments one or more vehicle surfaces may be mapped usingone or a combination or methods such as light and/or optical depthmapping and/or imaging of the vehicle surfaces and/or sensor mapping ofspatial position, orientation, motion and/or acceleration sensors and/orlocation sensors attached or embedded in each panel or display surfaceor section of the vehicle in order to generate a continuous map of thevehicle surfaces.

In some embodiments, the system includes a movement sensor to identify avelocity of the vehicle. The enhanced display content is further basedon the velocity of the vehicle.

In some embodiments, the system includes a projector. The processorutilizes the location and the velocity to a produce a second enhanceddisplay content and projects it on a fixed surface in proximity to thevehicle. In some embodiments, the fixed surface is a billboard or a busshelter.

An aspect of the present technology can provide an adaptive graphic userinterfacing system including a sensor component that can be configuredor configurable to dynamically map an environment in proximity to thesensor component to detect a user and a display candidate. A displayoutput component can be configured or configurable to provide displaycontent on or to the display candidate. At least one processing unit canbe operably connected or connectable to the sensor component, thedisplay component and at least one memory, the processing unit beingconfigured or configurable to determine a user's relational position tothe display candidate in real time, identify the display candidate andthe user, in real time, and continuously adapt the display content basedon the user's relational position to the display candidate in real-timeas the user moves in relation to the display candidate and/or thedisplay candidate moves in relation to the user.

In some embodiments, the display candidate can be selected from thegroup consisting of passive display systems, and active display systems.

In some embodiments, the display content can be selected from the groupconsisting of a graphical user interface, interactive media content,icons, images, video, text, interactive 3D objects, environments,buttons, and control affordances.

In some embodiments, the system further can include a microphone andspeaker component in operable communication with the processing unit,and being configured or configurable to receive and/or provide an audiosignal.

In some embodiments, the processing unit can be further configured orconfigurable to optimize the display content or a second display contentviewable by a second user based on a second user's relational positionto the display candidate or a second display candidate.

Another aspect of the present technology there is provided an imagingand display system comprising a sensor subsystem including at least onesensor configured or configurable to map an environment in proximity tothe system to identify a user position and a display candidate. Adisplay subsystem can include one of at least one projector configuredor configurable to project at least one display content on the displaycandidate, and a controller configured to control the display candidate.At least one processing unit can be operably connected or connectable tothe sensor subsystem, the display subsystem and at least one memory, theat least one processing unit being configured or configurable todetermine one or more characteristics of the display candidate,determine motion of an object in the environment, and configure thedisplay content based on one of the characteristics of the displaycandidate, and the motion of the object.

In some embodiments, the system can be a wrist mounted device with theprojector arranged to illuminate at least one of front and back of ahand wearing the unit, a wrist, forearm or finger of the hand, and thesensor is configured to detect a size or orientation of the hand wearingthe unit, a wrist, forearm or finger of the hand or entering a field ofview of the sensor.

In some embodiments, the processor can be continuously monitors the userhand to detect if a finger or a second object enters a predeterminedarea for determining interface inputs.

In some embodiments, the system can be eye glasses with the projectorbeing located to project onto at least one lens of the eye glasses, andthe sensor including at least one eye facing camera and at least oneoutward-facing camera.

In some embodiments, the outward-facing camera can continuously maps theenvironment, and the eye facing camera continuously monitors a field ofview for determining interface inputs.

In some embodiments, the system can be a virtual or augmented realityheadset.

In some embodiments, the system can include a plurality ofinterconnectable modules, with a first module including the sensor, anda second module including the projector.

In some embodiments, each module can include a communication portoperably connected with a communication port of an adjacent moduleconnected therewith.

In some embodiments, the system can further comprise one or moremicrophones and speakers.

In some embodiments, the system further can include an adapterconfigured to slidably mount the system to a track.

In some embodiments, the system can further comprise a base configuredto support at least a portion of a housing including at least one of thesensor subsystem, the display subsystem, and the processor.

In some embodiments, the base can include a touch screen interface, oneor more microphones and speakers, or at least one communication portconfigured to operably connect with a communication port of the system.

In some embodiments, the system can be associated with at least oneselected from the group consisting of a rearview mirror of a vehicle, adashboard of a vehicle, a drone, and a robot.

Yet another aspect of the present technology there is provided animaging and display system comprising a wrist mounted unit including atleast one sensor, and at least one projector. The sensor can beconfigured or configurable to detect a size or orientation of a handwearing the unit, a wrist, forearm or finger of the hand, or an objectheld in the hand or entering a field of view of the sensor. Theprojector can be arranged to illuminate at least one of front and backof the hand, wrist, forearm or fingers, or the object.

In some embodiments, the system can include at least one processing unitoperably connected or connectable to the sensor, the projector and atleast one memory, the at least one processing unit being configured orconfigurable to project display content by utilizing the projector basedon information received by the sensor or receive an input command byutilizing the sensor and detecting motion in the field of view of thesensor.

Still another aspect of the present technology there is provided animaging and display system comprising a head wearable device comprising.A sensor component can be configured or configurable to dynamically mapan environment in proximity to the sensor component to detect a user anda display candidate. A display output component can be configured orconfigurable to provide display content on or to the display candidate.At least one processing unit can be operably connected or connectable tothe sensor component, the display component and at least one memory, theprocessing unit executing an application to generate the display contentand to continuously adapt the display content based on a user'sposition.

In some embodiments, the display content can be projected onto a lens,at an eye of the user, or provided to an active display lens includingan embedded screen.

In some embodiments, the head wearable device can be an eye glasses unitcomprising at least one eye facing camera and at least oneoutward-facing camera, and at least one projector, the projector beinglocated to project onto at least one lens of the eye glasses, and theoutward-facing camera being configured to continuously map theenvironment, and the eye facing camera being configured to continuouslymonitor a field of view for determining interface inputs.

Yet still another aspect of the present technology there is provided animaging and display system comprising a wearable clothing comprising. Asensor component can be configured or configurable to dynamically map anenvironment in proximity to the sensor component to detect a user and adisplay candidate. A display output component can be configured orconfigurable to provide display content on or to the display candidate.At least one processing unit can be operably connected or connectable tothe sensor component, the display component and at least one memory, theprocessing unit executing an application to generate the display contentand to continuously adapt the display content based on a user'sposition.

A further aspect of the present technology there is provided an imagingand display system comprising. A display module including at least afirst and second communication ports, and at least one projectorconfigured or configurable to project at least one display content onthe display candidate. A sensor module can include at least onecommunication port, and at least one sensor configured or configurableto map an environment in proximity to the system to identify a userposition and a display candidate, the sensor module beinginterconnectable with the display module for communication between thefirst communication port of the display module and the communicationport of the sensor module. A light module can include at least onecommunication port, and at least one light source, the light modulebeing interconnectable with the display module for communication betweenthe second communication port of the display module and thecommunication port of the light module. At least one processing unit canbe operably connected or connectable to the sensor subsystem, thedisplay subsystem and at least one memory, the at least one processingunit being configured or configurable to determine one or morecharacteristics of the display candidate, determine motion of an objectin the environment, and configure the display content based on one ofthe characteristics of the display candidate, and the motion of theobject.

In some embodiments, the system can further comprise one or moremicrophones and speakers in operable communication with the processingunit.

In some embodiments, the system can further comprise an adapterconfigured to slidably mount the system to a track, the adapter beingextending from the sensor module.

In some embodiments, the sensor module can include a plurality ofsensors configured to provide 360 degree coverage in the environment,and the display module include a plurality projectors configured toprovide 360 degree coverage in the environment.

Another aspect of the present technology there is provided an imagingand display system comprising a sensor subsystem including at least onesensor configured or configurable to map an environment in proximity tothe system to identify an object position and a display candidate. Adisplay subsystem can include at least one projector. At least oneprocessing unit operably connected or connectable to the sensorsubsystem, the display subsystem and at least one memory. At least onegraphical user interface can be associated with the system andprojectable by the projector onto the display candidate or displayableon the display candidate, the graphical user interface comprising atleast one affordance configured or configurable to provide at least oneinput receivable and usable by the processing unit in controlling atleast one operational function. The display content can be based on oneof at least one characteristic of the display candidate detected by thesensor, and the motion of the object detected by the sensor.

Still another aspect of the present technology there is provided acomputing device comprising a sensor for mapping an environment inproximity to the device to identify a user position and a displaycandidate. A memory can contain an application coupled to a processor,the processor executing the application to generate display content. Theapplication can evaluate the display candidate and the user position andselecting the display candidate as a display surface. A display output,the application utilizing characteristics of the display surface and theuser position to modify the display content to produce enhanced displaycontent, the display output displaying an image of the enhanced displaycontent on the display surface.

In some embodiments, the application can continuously update the userposition and the display candidate to detect a perturbation in theenvironment, in response to the perturbation, the application modifyingthe enhanced display content to improve the image.

In some embodiments, the application can continuously update the userposition and the display candidate to detect a perturbation in theenvironment, in response to the perturbation, the application detectinga control action.

In some embodiments, the application can continuously update the userposition and the display candidate to detect a perturbation in theenvironment, in response to the perturbation, the application detectinga second display candidate and selecting the second display candidate asthe display surface.

In some embodiments, the display surface can be a passive surface andthe display output is a projector.

In some embodiments, the sensor can identify a second display candidate,the application selecting the second display candidate as a seconddisplay surface, wherein modifying the display content further comprisesdividing the display content between the display surface and the seconddisplay surface to produce second enhanced display content for thesecond display surface, the display output displaying a second image ofthe second enhanced display content on the second display surface.

In some embodiments, the computing device can comprise a plurality ofcomputing sub-devices networked together.

In some embodiments, the plurality of computing sub-devices can comprisetwo computing sub-devices worn on a wrist.

In some embodiments, the computing device can comprise a plurality ofcomputing devices networked together, wherein modifying the displaycontent further comprises allocating a first portion of the image to oneof the plurality of computing devices and a second portion of the imageto another of the plurality of computing devices.

In some embodiments, the system or device ca further comprise a networkinterface, the computing device can detect a network device anddetermining that the network device can create a second display surface,wherein modifying the display content further comprises dividing thedisplay content between the display surface and the second displaysurface to produce second enhanced display content for the seconddisplay surface, the computing device transmitting the second enhanceddisplay content to the network device for display on the second displaysurface.

In some embodiments, the computing device can be mounted on a part ofthe body of the user, the display surface comprising a body surface ofthe user.

In some embodiments, the part of the body can be a wrist and the bodysurface is a palm corresponding to the wrist.

In some embodiments, the part of the body can be a chest.

In some embodiments, the part of the body can be a head.

In some embodiments, the computing device can be incorporated into agarment, the garment comprising a plurality of light emitting elementsthat together comprise the display surface.

In some embodiments, the plurality of light emitting elements cancomprise OLED elements.

In some embodiments, the plurality of light emitting elements cancomprise a directional sensor that indicated to the processor the facingof the plurality of light emitting elements.

In some embodiments, the computing device can further comprise sportingequipment, the sporting equipment comprising a second sensor, the secondsensor in communication with the processor, the processor usinginformation received from the second sensor to produce the enhanceddisplay content.

In some embodiments, the device can be mounted in a fixed location inthe environment.

In some embodiments, the device can be mounted on a movable platform inthe environment.

In some embodiments, the computing device can be a wrist mounted deviceincluding one or more projectors for the display output and arranged toilluminate at least one of front and back of a hand wearing the unit, awrist, forearm or finger of the hand, or an object held in the hand, andthe sensor is configured to detect a size or orientation of the handwearing the unit, the wrist, forearm or finger of the hand, or theobject or entering a field of view of the sensor.

In some embodiments, the processor can continuously monitor the userhand to detect if a finger or a second object enters a predeterminedarea for determining interface inputs.

In some embodiments, the computing device can be eye glasses includingone or more projectors for the display output, the projectors beinglocated to project onto at least one lens of the eye glasses, and thesensor including at least one eye facing camera and at least oneoutward-facing camera.

In some embodiments, the outward-facing camera can continuously map theenvironment, and the eye facing camera continuously monitors a field ofview for determining interface inputs.

In some embodiments, the computing device can be a virtual or augmentedreality headset.

In some embodiments, the computing device can include a plurality ofinterconnectable modules, with a first module including the sensor, anda second module including the display output.

In some embodiments, each module can include a communication portoperably connected with a communication port of an adjacent moduleconnected therewith.

In some embodiments, the display output can include one or more lightemitters and projectors configured to project the display content ontothe display surface, and the sensor includes one or more sensors andcameras configured to detect the display surfaces.

In some embodiments, the computing device can further comprise one ormore microphones and speakers.

In some embodiments, the computing device can further comprise anadapter configured to slidably mount the computing device to a track.

In some embodiments, the computing device can further comprise a baseconfigured to support at least a portion of the computing device.

In some embodiments, the base can include a touch screen interface, oneor more microphones and speakers, or at least one communication portconfigured to operably connect with a communication port of thecomputing device.

In some embodiments, the computing device can be associated with arearview mirror or a dashboard of a vehicle.

In some embodiments, the computing device can be associated with a droneor a robot.

Still another aspect of the present technology there is provided animaging and display device comprising an imaging system capturing asequence of images in a field of view. A movement sensor can output aplurality of movement data indicative of movement of the device withinan environment. A location sensor can output a plurality of locationdata indicative of a location of the device. A processor module can becoupled to the imaging system, the movement sensor, and the locationsensor, the processor module receiving the sequence of images, theplurality of movement data, and the plurality of location data, theprocessor module analyzing the sequence of images to determine if acontrol object in the field of view is in motion. The processor modulecan analyze the plurality of movement data and determining anorientation of the device. The processor module can analyze theplurality of movement data and the plurality of location data anddetermining the location of the device. The processor module candetermine if an event has occurred.

In some embodiments, the processor module can further determine that theevent is a control gesture performed by the control object, theprocessor module then executing a routine indicated by the controlgesture.

In some embodiments, the processor module can further determine that theevent is a perturbation of the location of the device or the orientationof the device and optimizing a display output of the device.

In some embodiments, the processor module can further determine that theevent is a change in an environment of the device, the processor modulereconfiguring the device for the environment.

In some embodiments, the reconfiguration of the device can comprisechoosing a new display surface.

In some embodiments, the device can be a wrist mounted device includingone or more projectors configured to display an image on at least one offront and back of a hand wearing the unit, a wrist, forearm or finger ofthe hand, or an object held in the hand, and the sensor is configured todetect a size or orientation of the hand wearing the unit, the wrist,forearm or finger of the hand, or the object or entering a field of viewof the sensor.

In some embodiments, the processor can continuously monitor the userhand to determine if the event has occurred.

In some embodiments, the device can be eye glasses including one or moreprojectors configured to project an image onto at least one lens of theeye glasses, and at least one eye facing camera and at least oneoutward-facing camera.

In some embodiments, the outward-facing camera can continuously map anenvironment exterior of the eye glasses, and the eye facing cameracontinuously monitors a field of view for determining interface inputs.

In some embodiments, the device can be a virtual or augmented realityheadset.

In some embodiments, the device can include a plurality ofinterconnectable modules, with a first module including one of theimaging sensor, movement sensor or the location sensor, and a secondmodule including a display output.

In some embodiments, each module can include a communication portoperably connected with a communication port of an adjacent moduleconnected therewith.

In some embodiments, the device can include one or more light emittersand projectors configured to project display content onto a displaysurface, and the imaging sensor includes one or more sensors and camerasconfigured to detect the display surface.

In some embodiments, the device can further comprise one or moremicrophones and speakers.

In some embodiments, the device can further comprise an adapterconfigured to slidably mount the computing device to a track.

In some embodiments, the device can further comprise a base configuredto support at least a portion of the device.

In some embodiments, the base can include a touch screen interface, oneor more microphones and speakers, or at least one communication portconfigured to operably connect with a communication port of the device.

Yet still another aspect of the present technology there is provided acomputing device displaying an image, the device comprising a sensor formapping a plurality of positions corresponding to a plurality ofviewers, each of the plurality of positions comprising a viewer locationand a viewer orientation in an environment. A memory can contain anapplication coupled to a processor, the processor executing theapplication to generate display content on a plurality of surfaces. Theapplication can evaluate the plurality of positions and customizing thedisplay content bases on the plurality of surfaces, the applicationdisplaying the display content on the plurality of surfaces.

In some embodiments, the environment can be a vehicle.

In some embodiments, the device optimizes the display content for eachof the plurality of viewers can be based on the role of each of theplurality of viewers in an operation of the vehicle.

In some embodiments, the device can optimize the display content foreach of the plurality of viewers based on the position of each of theplurality of viewers in an operation of the vehicle.

In some embodiments, the position of each of the plurality of viewerscan comprise a seating arrangement.

In some embodiments, the processor can optimize the display content foreach of the plurality of viewers based on a physical characteristic ofeach of the plurality of viewers.

In some embodiments, the physical characteristic can comprise height.

In some embodiments, the system or device can further comprise aplurality of light sensors, the processor coupled to the plurality oflight sensors and receiving ambient light information from the pluralityof light sensors in proximity to the plurality of users, the processoroptimizing the display content for each of the plurality of viewersbased on the ambient light information.

In some embodiments, the system or device can further comprise anexternal light sensor, the processor coupled to the external lightsensor and receiving external light information from the external lightsensors, the processor utilizing the external light information whenoptimizing the display content for each of the plurality of viewers.

In some embodiments, each of the plurality of viewers can have apersonal mobile device providing an identity of the plurality of viewersto the processor, the processor optimizing the display content for eachof the plurality of viewers based on the identity of each of theplurality of viewers.

In some embodiments, the device can include a plurality ofinterconnectable modules, with a first module including the sensor, anda second module including at least one projector configured to displaythe display content.

In some embodiments, each module can include a communication portoperably connected with a communication port of an adjacent moduleconnected therewith.

In some embodiments, the device can include one or more light emittersand projectors configured to project the display content onto thesurfaces, and the sensor includes one or more sensors and camerasconfigured to detect the surfaces.

In some embodiments, the device can further comprise one or moremicrophones and speakers.

In some embodiments, the device can further comprise an adapterconfigured to slidably mount the computing device to a track.

In some embodiments, the device can further comprise a base configuredto support at least a portion of the computing device.

In some embodiments, the base can include a touch screen interface, oneor more microphones and speakers, or at least one communication portconfigured to operably connect with a communication port of thecomputing device.

A further aspect of the present technology there is provided a computingdevice incorporated in a vehicle, the device comprising a sensor formapping an environment in proximity to the vehicle to identify alocation of the vehicle. A memory can contain an application coupled toa processor, the processor executing the application to generate displaycontent. A display surface can be an exterior surface of the vehicle,the application evaluating the location and modifying the displaycontent to produce enhanced display content based on the location, theenhanced display content being displayed on the display surface.

In some embodiments, the display surface can comprise a vehicular bodywrap.

In some embodiments, the vehicular body wrap can comprise OLED elements.

In some embodiments, the system or device can further comprise amovement sensor to identify a velocity of the vehicle, the enhanceddisplay content further bases on the velocity of the vehicle.

In some embodiments, the system or device can further comprise aprojector, the processor utilizing the location and the velocity to aproduce a second enhanced display content and projecting it on a fixedsurface in proximity to the vehicle.

In some embodiments, the fixed surface can be a billboard.

In some embodiments, the fixed surface can be a bus shelter.

In some embodiments, the device can be associated with a rearview mirroror dashboard of the vehicle.

Still another aspect of the present technology there is provided amethod of providing display content to a display candidate utilizing animaging and display system. The method can comprise the steps of mappingan environment in proximity to an imaging and display system to identifya user position and a display candidate utilizing at least one sensor.Projecting at least one display content on the display candidateutilizing at least one projector, or controlling the display candidateto display the display content utilizing a controller. Determining oneor more characteristics of the display candidate utilizing at least oneprocessing unit operably connected or connectable to the sensor, thedisplay and at least one memory. Determining motion of an object in theenvironment. Configuring the display content based on one of thecharacteristics of the display candidate or the motion of the object.

Still yet another aspect of the present technology there is provided anadaptive graphic user interfacing system can comprise a set of devicesin communication with each other and configured to work together toadapt continuously to changing sets of input and output devices,changing sets of users, changing computation environments, and changingusage needs.

The foregoing and additional aspects and embodiments of the presentdisclosure will be apparent to those of ordinary skill in the art inview of the detailed description of various embodiments and/or aspects,which is made with reference to the drawings, a brief description ofwhich is provided next.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other advantages of the disclosure will becomeapparent upon reading the following detailed description and uponreference to the drawings.

FIG. 1 illustrates an embodiment of an AGUI system projecting a displayon the ceiling in the field of view of a viewer.

FIG. 2 illustrates how the AGUI system adapts as a viewer's positionchanges, displaying on a vertical wall with additional, contextsensitive information.

FIG. 3A illustrates how gestures can be used to control the AGUI system.

FIG. 3B illustrates how the AGUI system can dynamically select newsurfaces as they become available.

FIG. 4A illustrates how an AGUI system maps and identifies a user usingstructured light imaging.

FIG. 4B illustrates how an AGUI system maps and identifies a user usingcolor and optical light imaging.

FIG. 5A illustrates how structured light mapping or depth mapping may beused to detect a user.

FIG. 5B how optical light can be used to map a user and identifysurfaces that can be an interface.

FIG. 6A-6C illustrate the use of a wrist mounted AGUI device the maps auser's surfaces, selects display surfaces, and adapts when the surfacesmove.

FIG. 7A-7C illustrates alternative ways of dividing up a hand intodisplay surfaces.

FIG. 8A-8C illustrates alternative ways of dividing up a hand and wristinto display surfaces.

FIG. 9A-9C illustrates alternative ways of dividing up the back of ahand and forearm into display surfaces.

FIG. 10A-10B illustrates how a selection in one display can initiate arelated display on another surface.

FIG. 11A-11B illustrates how display content can be projected from avariety of sources.

FIG. 12A-12B illustrates how a illustrates how display content can beprojected from a variety of sources for both a main display and relateddisplay.

FIG. 13A-13B illustrates how display content can be projected from avariety of sources for both a main display and related display.Additionally, there is a third active display on a wrist mounted AGUIdevice.

FIG. 14A-14B illustrates how an external AGUI device may use a handheldobject can be detected and used as a display surface.

FIG. 15A-15B illustrates how a body mounted AGUI device may use ahandheld object can be detected and used as a display surface.

FIG. 16A-16B illustrates how an AGUI device may detect and use ahorizontal surface as a display surface.

FIG. 17A-17B illustrates how an AGUI device may detect and use avertical surface as a display surface and how gestures or touch can beused to manipulate the GUI.

FIG. 18A-18C illustrates a wearable AGUI projector; lapel or headmounted. Also illustrated is how the two AGUI projectors can worktogether.

FIG. 19A-19B illustrates how an AGUI system can project two differentinterfaces on the same object.

FIG. 20-A20-B illustrates AGUI enabled glasses that provide a heads-upbinocular display.

FIG. 21A-21B illustrate the use of an AGUI virtual reality headset beingused to manipulate a virtual object using multipoint controls in space.

FIG. 22A-22B illustrate how an AGUI system may adapt display elements tothe shape of surface.

FIG. 23A-23D illustrate how an AGUI system may adapt display content tothe shape and size of displays.

FIG. 24A-24B illustrates how an AGUI system may use clothing withorganic flexible display elements embedded in fabric as a display.

FIG. 25A-25C illustrates another example of the use of clothing withorganic flexible display elements embedded in the fabric as a display.

FIG. 26A-26C illustrates an example of the use of clothing with organicflexible display elements embedded in the fabric as a display where theindividual clothing pieces are used together to present an individualdisplay.

FIG. 27 illustrates multiple AGUI devices networked in an environment,detecting and interacting with a user in real time.

FIG. 28A-28D illustrates two users interacting with a single AGUIsystem.

FIG. 29A-29C illustrates the use of an AGUI system in a kitchenenvironment including AGUI enabled appliances.

FIG. 30A-30D illustrates the use of an AGUI system to visualize theposition of virtual reality furniture in a room.

FIG. 31A-31B illustrated the use of an AGUI system in a virtual realitygame including an augmented reality virtual environment.

FIG. 32A-32D illustrates an AGUI enabled vehicle where the surface ofthe vehicle comprises active display surfaces.

FIG. 33 illustrates AGUI enabled vehicles operating with AGUI enabledbuildings in an environment.

FIG. 34 illustrates a user in an environment with an AGUI systemcomprising moving and stationary AGUI devices.

FIG. 35A-35B illustrates the use of an AGUI system in a vehicle wherethe interior and occupants are mapped and displays identified and used.

FIG. 36A-36C illustrates the use of an AGUI system in a vehicle todisplay information customized for a driver.

FIG. 37A-37B illustrates the use of an AGUI system in a vehicle interiorillustrating a number of possible AGUI projectors.

FIG. 38A-38B illustrated the use of an AGUI system in an officeenvironment.

FIG. 39A-39C illustrates the use of an AGUI system in a hospitaloperating room environment prior or after an operation.

FIG. 40 illustrates the use of an AGUI system in a hospital operatingroom to provide information during an operation.

FIG. 41A-41B illustrated a drone based AGUI system.

FIG. 42A-42B illustrates a drone based AGUI system that may be used todeliver packages.

FIG. 43 illustrates a drone based AGUI system projecting instructions ona surface.

FIG. 44A-44B illustrates a rolling AGUI device that may moveintelligently through its environment.

FIG. 45 illustrates a combination of AGUI projectors, flying, moving orstationary, interacting with multiple viewers.

FIG. 46A-46B illustrates a track mounted AGUI device with a modulararchitecture.

FIG. 47A-47C illustrates different plugin pieces and interfaces modulesfor an AGUI system. Microphones and sound. Combined with light source.Power port options (light bulb). Desk or track light. Different types oflight interfaces. Hole is data and power port. Base or docking system.

FIG. 48A-48B illustrates an AGUI device with an associated, non-AGUIdevice.

FIG. 49A-49C illustrates a modular base for an AGUI device.

FIG. 49D is a diagrammatic representation of a plenoptic image systemaccording to an embodiment that can be utilized in any one or moreembodiments of the present technology AGUI system or device;

FIG. 49E is a diagrammatic representation of a plenoptic image systemaccording to an embodiment that can be utilized in any one or moreembodiments of the present technology AGUI system;

FIG. 50 illustrates a block diagram of the components of an AGUI system.

FIG. 51 illustrates a flow chart of how an AGUI may detect and useavailable surfaces for displays.

FIG. 52 illustrates a flow chart of how an AGUI system can select asurface to use as a display.

FIG. 53 illustrates the use of a network of AGUI devices where thecentral hub is also an AGUI display, interfacing, computing,communications and networking device.

FIG. 54 illustrates the use of a network of AGUI devices connected to acentral AGUI HUB where the HUB is an Internet or other network basedplatform and/or operating system that lacks a physical device other thanthe devices and systems that are connected to the HUB; and

FIG. 55 is a diagrammatic representation of an embodiment of a machinein the form of a computer system.

FIG. 56 is a diagrammatic representation of an embodiment of a generalAGUI operation of an AGUI device

REFERENCE NUMERALS IN DRAWINGS

-   1 Computer System-   5 Processor or multiple processors-   10 Main Memory-   15 Static Memory-   20 Bus-   30 Input Device(s)-   35 Display-   37 Drive Unit-   40 Signal Generation Device-   45 Network Interface Device-   50 Computer or Machine-Readable Medium-   55 Instructions and Data Structures-   60 General Synopsis of the AGUI operation-   62 Core Processing-   64 Dynamic Display-   66 Dynamic GUI-   68 Change Mechanism-   70 Setup-   Plenoptic System-   72 Objective Lens-   74 Microlens Array-   76 CMOS Sensor-   78 Focal Plane of the Objective Lens-   100 (Wearable Device/Projection Interfacing Systems)-   101 Wrist mounted Projection Interfacing System-   102 Wrist Module Screen-   106 Wrist Straps-   108 Light Emitters-   110 Light Sensors-   112 Cameras/Camera Array (Wrist Console)-   114 Biometric IR/Light Sensors (Wrist Console)-   115 Projector-   116 Haptic feedback sensors (Wrist Console)-   120 Head mounted Projection interfacing system-   130 Chest mounted projection interfacing system-   140 Heads-up projection interfacing system-   150 Heads-up Display Glasses-   152 Heads up display glass—Left lens-   154 Heads-up display glasses—Right Lens-   156 Heads-up display glasses—lens projector-   158 Heads-up display glasses—eye-facing camera-   160 Outward-facing cameras-   168 Virtual Reality Headset-   170 Augmented Reality Headset-   172 Augmented Reality Headset Visor-   200 (Desktop/Mounted Projection Interfacing/Multimedia Hub)-   201 Multimedia Hub—mapping Imaging, Interfacing and adaptive display    System-   202 Light Emitters & Projector Array-   204 Light Sensors & Camera Array-   206 Microphone & Speaker Array-   208 Primary Multimedia Hub module-   210 Microphone/Speaker Module-   212 Power/Data Port (in any module)-   214 Adapter for use in screw-in light fixtures-   216 Module with light sensors and camera array-   220 Light Module (LED or Light Bulb Cover)-   222 Alternate Light Module—Halogen/Light Bulb Adapter/Port-   223 Standard halogen bulb/LED bulb-   224 Track-   225 Track mount for device-   226 Track Lighting Port-   228 Touch Screen Display/Interface-   240 Base Unit-   242 Base Unit—Charging, USB/Data Port-   244 Base Unit—Phone/Cable/Internet Port-   246 Base Unit—Microphone/Speaker Array-   248 Base Unit—Touch Screen/Button Array-   250 Table top stand with articulating arm-   252 Table top stand—straight post-   300 (Mapping, Imaging, Zoning & Interfacing Systems)-   302 Performing Light and/or Optical Depth & Color Mapping & Imaging-   304 Performing a Light Scanning Operation using one or moving beams    of light-   306 Performing Depth Mapping using a structured light imaging system-   308 Performing Depth and Color mapping and Imaging using a camera    array-   310 Identifying the precise position of the joints and creases of    the hand and body-   312 Generating a functional rigging of the hands and body-   314 Create a fully functional computer model of the hand and body    capable of being animated in real time-   320 Surface Zoning—Identifying objects on a surface and identifying    optimum display zones around mapped objects-   322 Surface Zoning —Assigning a Graphic UI and adapting content,    sizing, format and layout of UI to zones around mapped objects-   324 Surface Zoning—Assigning a Graphic UI to an object, device or    surface-   330 Hand Zone-   332 Palm zone-   334 Finger Zones-   336 Sub Zones-   338 Arm Zone-   340 Palm and Fingers-   342 Back of Hand and Fingers-   344 Inside of Arm-   346 Back of Arm-   348 Surface Zoning—assigning a Graphic User Interface around mapped    objects-   348 Surface Zoning—assigning a Graphic User Interface to an object,    device, or surface-   350 Depth Mapping & imaging of an Object-   352 Depth Mapping & Imaging of Multiple Surfaces-   354 Depth Mapping & Imaging of a Room-   360 Projection Mapping and Display-   380 3D Mapping & Interfacing-   382 External Mapped Object-   384 External Networked Device-   386 External Networked Display/Device/Computer-   388 External Mapped Projection Surface-   390 External Mapped Display-   392 External Mapped Equipment-   400 Interfacing Systems-   (Touch, gesture, body motion, eye motion, voice command, audio and    haptic response)-   401—Multi-point touch interfacing-   402—Hand & fingers-   404—Arm-   410 Multi-point gesture interfacing-   420 Input hand-   422 Display hand-   (input and display hands can be interchangeable, or one hand can be    both at the same-   time or sequentially)-   430 Body motion-   435 Facial motion and expression control-   440 Eye motion command & point of focus-   450 Voice command-   460 Haptic interface-   500 (AGUI Display Systems & Methods)-   501 Adaptive Graphic User Interface-   502 Projected Adaptive Graphic User Interface-   503 Control menu/Interface-   504 Keyboard/Keypad on a table/surface-   505 Sub-menu/Interface-   506 Projected display interface on hand-held device or object-   510 Projected interactive multimedia content from remote projector-   512 Projection from wrist device-   515 Organic adaptive user interface-   516 Organic adaptive multimedia content-   520 Selected icon, menu item or tab-   530 interactive icons-   532 interactive tabs/menu items-   534 interactive multimedia content-   536 video conf.-   538 virtual buildings-   540 Binocular heads up display-   545 Augmented or virtual graphic user interface (assigned to a    virtual object/actual-   object/empty space/depth of field)-   550 Vehicle Head-up display/Interface-   552 Projected display/Interface onto a vehicle surface-   560 Projected patent vital statistics (e.g. EKG)-   562 Projected patient internal organ virtual representation-   600 (Clothing—Organic Adaptive Display & Interfacing)-   601 Adaptive Display Clothing-   602 Adaptive Display Jacket-   603 Adaptive Display Pants-   604 Adaptive Display Dress-   605 A Graphic Display Assigned to-   610 Virtual-   700 (Physical Objects, Displays, Devices, Vehicles & Drones)-   702 Laptop-   704 Tablet-   706 Smart Phone-   708 Smart Watch—Round Display-   710 Smart Watch—Rectangular Display-   712 Note Pad-   714 Game Ball—Toy-   718 WebCam-   720 TV Screen/Mounted TV/Display-   722 Real Entertainment and-   724 Sound System/Speakers-   730 Dining table dinnerware, etc.-   740 Table-   750 UAV/Drone-   760 Robot—Robotic vehicle/UTV? (unmanned terrestrial vehicle)    wheeled/track vehicle-   (Virtual Objects, Displays & Interfaces)-   800 Virtual representation of man-   802 Virtual 3D object-   810 Virtual Furniture-   820 Virtual TV/Display-   822 Virtual Entertainment and Sound System-   824 Virtual Speakers-   830 Virtual footballer-   831 Virtual footballers mask-helmet-   832 QR Code-   834 Virtual Baseball Pitcher-   835 Virtual Baseball-   836 Virtual Baseball Batter-   838 Virtual Baseball Catcher-   840 Projected instructions-   842 Projected directions-   844 Projected identification symbols/words/imagery-   845 Displayed Text/Words-   850 Projected food items for reorder/cooking prep.-   852 Projected delivery truck-   854 Projected meal selection-   860 Virtual golf club-   862 Virtual golf ball-   864 Virtual golf course-   866 Virtual trees-   868 Virtual sand traps-   870 Virtual Landscape-   Interior/Exterior Vehicle Mapping, Imaging, Interfacing & Adaptive    Display)-   900 Car/Vehicle/Taxi-   901 Interior Vehicle Multimedia Hub—Projection Mapping, Imaging,    Interfacing and-   Adaptive Display System-   902 Rearview Mirror Vehicle Windshield Head-up Display Projection    Interfacing System-   904 Dashboard mounted Windshield Head-Up Display Projection    Interfacing System-   906 Vehicle Windshield Head-up Display (Embedded Glass or Projection    Display)-   908 Front Driver Window Display (Embedded Glass or Projection    Display)-   910 Front Passenger Window Display (Embedded Glass or Projection    Display)-   912 Rear Passenger Window Display (Embedded Glass or Projection    Display)-   914 Vehicle Interior Mapping, User Identification & Display/Content    Selection-   920 Bus-   922 Bus Display Panels-   924 Bus Side Window Displays-   930 Bus Stop-   932 Bus Stop Display Panels/Screens-   940 Buildings-   942 Building Display Windows/Panels/Screens-   945 Street Display/Projection-   Exterior Vehicle Mapping, etc.-   950 Exterior light imaging and camera array on vehicle+projectors-   952 Vehicle wind shield-   954 Vehicle Windows (side, rear)-   956 Vehicle body panels-   958 Vehicle Digital Wheel Display-   960 Taxi Top Digital Advertisement Display-   Illustrated/Mapped People/Users-   1000—Person (User/Wearer)-   1002—User Male (Person)-   1008—Rigged User-   1010—Fully Modeled and Animated User model-   1020—Person Female (Person)-   1021—3D mapped female-   1022—Person Male Child-   1024—Person Female Child-   1026—Person—Patient-   1028—Medical personnel/Surgery personnel/Doctor/Surgeon-   1030—Group of People-   1040 Professional Golfer-   1042 Sensors on or embedded in clothing/Shoes-   1044 Sensors embedded into equipment-   1100 (Exterior Body Parts and Systems)-   1102—Hand-   1104—Arm-   1104—Inside of Hand (Palm)-   1106—Back of Hand-   1108—Fingers-   1110 —Arm-   1112—Inside of Arm-   1114—Back of Arm-   1150 (Interior Body Parts, Organs and Systems)-   1152 Virtual Internal Body Map-   1154 Nervous System-   1156 Endocrine System-   1158 Brain Machine Interface-   1160 Prosthetics (external)-   1162 Implanted Device (Pacemaker)-   1164 Skeletal System-   1166 Muscular System-   1168 Integumentary System-   1170 Cardiovascular System-   1172 Respiratory System-   1174 Lymphatic System-   1176 Digestive System-   1178 Urinary System-   1180 Reproductive System-   1182—Skin-   1200 Physical Rooms, Environments, Objects, Devices & Surfaces-   1200 A residential Living Space-   1202 3D Mapped Environment-   1210 Entry Way-   1212 Ceiling-   1214 Wall-   1216 Floor-   1220 Bedroom-   1222 Bed-   1230 Kitchen-   1232 Refrigerator-   1234 Cabinet Display-   1235 Oven-   1236 Microwave-   1237 Coffee Machine-   1238 Counter Top-   1240 Living Room-   1250 Dining Room-   1252 Dining Room Table-   1254 Office Cubicle-   1256 Office Desk-   1258 Office Cubicle Front Wall-   1260 Office Cubicle Side Wall-   1270 Operating room at Hospital/Clinic-   1272 Privacy screen-   1274 Overhead lighting with integrated projection and scanning    system-   1300 (Networking & Communications)-   1302 AGUI Internet HUB/Server/Platform-   1303 Cell Tower/Wireless Communications-   1304 Home Systems & Appliances-   1306 AGUI Device Group Network-   1308 Desktop, Enterprise and Mobile Computing Devices and    Entertainment Systems-   1310 Wearable, AR, VR and Mixed Reality Computing, Display and    Interfacing Systems-   1312 Vehicles, Robotics and Transportation Systems-   1314 Business, Retail, Buildings, Billboards, Indoor and Outdoor    Display Systems-   1400 Operating & Intelligence Systems & Processes-   1402 His . . .-   1404 Hers . . .-   1500 Patent Diagrams

While the present disclosure is susceptible to various modifications andalternative forms, specific embodiments or implementations have beenshown by way of example in the drawings and will be described in detailherein. It should be understood, however, that the disclosure is notintended to be limited to the particular forms disclosed. Rather, thedisclosure is to cover all modifications, equivalents, and alternativesfalling within the spirit and scope of an invention as defined by theappended claims.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

It has been identified that graphical user interfaces are often designedfor a specific device, display type or display surface with limitedadaptability or interoperability between one device, display type orsurface and another. It has been identified that a drawback of existingtechnology is that dynamic surfaces, environments and ambient lightingconditions used by projectors and active displays are difficult tomanage when they change in size, shape form, color, brightness,reflectivity, orientation, or environmental conditions such as ambientlight and other variables. Systems are not able to adapt to displays anddisplay surfaces entering and exiting an environment in an unpredictablemanner or when the displays are not fixed in size, shape, andorientation or when moving or transitioning from one display system orsurface to another. Furthermore, graphic user interfaces are oftenlimited to a single device, display type or surface and not adaptable toa dynamic environment and mix of displays and projection surfaces. Ithas been further identified that there is need for an adaptive graphicuser interface (AGUI) that can take into account a variable number ofdisplays and display surfaces that may vary in size, shape and type,with the AGUI dynamically adapting to the environment to allow a user tobetter interact with the system.

1. ARCHITECTURE AND EXAMPLE DEPLOYMENT 1.1. Example Embodiments

Embodiments of the AGUI may take several forms building on the keyconcepts and features of the system.

Embodiments of the present technology comprise hardware, software, andmethods to implement an AGUI (Adaptive Graphical User Interface) systemthat allows a user to interface with a computerized system in a dynamicand changing environment. Components of an AGUI system can operateindividually or collectively on a projector, PC, laptop, tablet, phone,television, watch, glasses, headset, clothing, vehicle or othercompatible hardware, display system or surface. AGUI is an operatingplatform that can operate on one or more independent, paired ornetworked devices. The AGUI operating platform may be embedded orinstalled on one or more devices, appliances, homes, enterprises,vehicles and/or other systems downloaded as a program, platform orapplication on one or more devices and/or other systems or operate viaone or more servers as an Internet or other network based HUB in which amultitude of devices and/or other systems may connect, interface and/orbe managed by the AGUI system or any combination of these installationand/or operating, networking and interfacing methods.

According to one embodiment of the present technology, the AGUI systemmay comprise one or more ceiling, wall, floor, desktop or other surfacemounted projectors. The AGUI projectors may operate independently or maybe wired or wirelessly networked together and operate as a group tosense the environment, implement displays and interfaces, connect,interface with and control networked devices and systems and detect usercommands.

According to another embodiment of the present technology, the AGUIsystem may comprise one or more handheld, mobile, desktop, mounted orother displays such as phones, PC's, monitors, laptops, tablets,televisions and other active display and interfacing systems, screensand surfaces, such as touchscreens, motion and gesture interfaces LED,OLED, LCD, Plasma and other display types. The AGUI system may adapt agraphic user interface or other multimedia content to the active displaydevice and/or surface by receiving precise size, shape, pixel or otherdimensional and display specifications from the device and then adaptingand optimizing the graphic user interface or other multimedia contentfor the AGUI display device and/or surface. When coordinating contentacross two or more networked active display devices and/or surfaces theAGUI system may map or assign specific content to each device, mayassign one device as the primary or master device and other devices asthe control or slave devices and/or the AGUI system may organized and/orassign different functions and interfaces to each networked activedisplay device and/or surface or the AGUI system may group the devicesand/or surfaces and adapt content across some or all of the activedevices and/or surfaces.

The AGUI system may also map content across both active and passivedisplay surfaces for example if a TV screen is attached to a wall anAGUI projection.

According to another embodiment of the present technology, the AGUIsystem may comprise one or more wearable devices, projectors or head-updisplay systems including but not limited to watches, headsets, glasses,visors, helmets, windshields and other wearable devices and head-updisplay systems. Such as a wrist or other body worn AGUI Device withmicro-projectors projecting a display and user interface on a user'shand, fingers, arm, body and/or surrounding objects and surfaces.micro-projection device may contains sensors and determines the bounds,shape, location, and orientation of the user's hand fingers, arm, bodyand surrounding surfaces and environment. The system also maps the hand,fingers, arm, body and surrounding objects and surfaces so as todetermine the multiple surfaces that a graphic user interface, text,image, video and/or other multimedia content may be projected on as wellas to map the interfacing hand, fingers or other interfacing device orobject in which to control the projected graphic user interface and/orother multimedia content and display. The user interface is thenprojected on the hand and may be manipulated by touching the parts ofthe displayed surface with a hand, finger, pointer, or another device.In a similar manner, the AGUI system may also use fingers, arms, otherbody surfaces, and other surrounding surfaces such as a desk or wall toproject on. A user is equipped with an AGUI system that comprises amicro-projector on one or both wrists or other with which to interactwith their environment. User interfaces may be projected on any surfacesuch as the hand and fingers, the arm or other body surface, on a deskor other furniture, appliance or other objects, on a wall, floor,ceiling or other surface and allows the user to control devices in theroom including the laptop computer, table, television, audio system, andlights. As the user moves within a room, the system tracks where theyare, their location and velocity, and adapts the AGUI for this.

The wearable projectors may operate in concert with surrounding devicesand active display system such as PC's, monitors, laptops, phones,tablets, televisions, furniture, appliances, vehicles and othercomputing and/or display devices and systems to form an AGUI Groupsystem. The AGUI Group system scans the environment to detect the userand identify the AGUI projectors, PC, laptop, tablet, TV or other activedisplay devices, screens and/or surfaces. The AGUI projectors and activedisplays are able to communicate with each other to implement the AGUIsystem. The micro-projectors work in tandem with connected devices andsurfaces to project an interactive graphic user interface and/ormultimedia content displayed on any of the body, desktop, ceiling, wall,floor, furniture, appliances, PC, TV or other active or passive displaysurface for the user to interact with. The AGUI system may also adaptany type of 2D or 3D graphic user interface including icons, keyboard,text, images, video, interactive graphics or other multimedia content toall passive and active display systems and surfaces. The user mayinterface with the interactive graphic user interface and/or otherdisplayed multimedia content as they would on a physical keyboard orother active touchscreen, motion, gesture or other 2D, 3D ormulti-dimensional computing display and/or environment. Themicro-projectors detects and maps the user's hand, fingers, and bodymotion and spatial position and determine which icons or otherinterfacing control by the user. The user may also use finger, hand,body, eye facial gesture and movements, voice command or other usercommands and inputs, which are detected by sensors in or connected tothe micro-projector devices, as a pointing or control device foraconnected PC, laptop, phone, tablet, TV, appliance, vehicle or otherconnect device or system. While typing and/or interfacing the user'shands and body will move, causing the micro-projectors to move. Despitethese movements or perturbations, the micro-projectors will continue todisplay the keyboard or other graphic user interface or multimediadisplay as a stable image in the assigned position on the table. TheAGUI system continuously and dynamically adapts to the environment,including the top or bottom of an arm, hand, fingers, and surroundingsurfaces such as a wall, floor, ceiling, furniture, appliance or otherdevice or object depending on the orientation and position of theseobjects, the relations to the viewer, and the type of application. Themicro-projectors will also automatically ensure that the image of thevirtual keyboard or other graphic user interface, image, video or othermultimedia content is optimally displayed and undistorted no matter theposition of the user's hands and the angle the micro-projectors makewith the table. The micro-projectors will utilize internal sensors toscan and/or map and determine the surface characteristics (color,texture, transparency, reflectivity) of the table, adjust the outputimage, and project the virtual keyboard image to ensure it is easilyvisible. The micro-projectors may also be connected via a network to aphone, PC, laptop computer, tablet, TV, appliance, headset, glasses,vehicle or other connected device or display and determine whatinformation to display on each device as well as to which device ordevices the virtual keyboard, graphic or other user interface inputshould be sent to.

Alternatively the AGUI Device may be a head up display system such as apair of glasses, a headset, visor or other wearable head-up displaywhich may incorporate one or more pico projectors to project a microdisplay onto one or more lenses in front of the eyes, a visor or otherhead-up display surface. The AGUI system may also adapt a graphic userinterface for projecting an image directly into the eyes, and/or mayadapt content for an active head-up display such as an LED, OLED, LCD orother screen or active display system. The AGUI system may also drive oroperate in coordination with a brain machine interface and/or bodymachine interface in which interactive graphic and/or other multimediacontent is sent directly to one or more receptors implanted in the brainor body, or to a prosthetic such as prosthetic eye, ear or other sensorysystem or otherwise assigned to the brain or body. In these scenariosthe adaptive graphic user interface may be controlled using eye motion,brain activity or when incorporating other sensors such as but notlimited to light, optical, audio, motion, location and biometric sensorsmay also include touch, gesture, motion, emotion and other input andinterfacing methods.

According to another embodiment of the present technology the AGUIsystem may comprise organic, flexible, malleable or other non-fixed ornon-static form factor devices and display systems such as fabric,clothing, shoes, hats, helmets and other equipment, furniture, home,enterprise and vehicle upholstery and other objects, devices, displaysystems and surfaces which may naturally, mechanically or electronicallychange in shape, size or form. The AGUI system may continuously generatea depth map of an organic display system using light and/or opticalimaging of the surface, using embedded spatial position, orientation,acceleration, location or other sensors in order to generate a pointcloud and depth map of the surface for the purpose of continuouslyadapting a graphic user interface to the adaptive display surface.

According to another embodiment of the present technology the AGUIsystem may comprise one or more vehicles and interior and exteriorvehicle display systems and surfaces.

A) Internal Vehicle AGUI Device/Display System

B) Exterior Multi-Panel/Surface Content Mapping, Zoning and interfacingsystem

C) Multi-Vehicle Interfacing System/One or more viewers—Adapting contentbased on relational/moving position of one vehicle to another and basedon the relational viewing position of one or more viewers

D) Multi-Vehicle/and Outdoor Display Surfaces (Building/Roads/BusStops/Billboards etc.) Location/Relational/Contextual and Event specificcontent management

The AGUI system is centered around an AGUI controller device or acollection of controller devices communicating with each other and withexternal devices via a wired or wireless network. A wrist mounted deviceis merely one or more embodiments of the AGUI controller device. As ismade readily apparent from the description of embodiments of the presenttechnology set forth herein, the AGUI Device has many other embodimentsand aspects and is in no way limited to the wrist mounted device. It mayalso be a device fixed to a location such as to the ceiling or on adesk. The AGUI device may also be in motion, mounted on a drone orincorporated in a vehicle. It may be adapted to the interior or exteriorof a car, bus, plane, boat or other vehicle. In some embodiments, aheads-up display is adapted to the windshield of a car or other vehicleand other interactive media content may be adapted to the other windowsor the backs of chairs, doors and other display surfaces on the insideof a vehicle, which may be assigned based on the position of one or morepassengers inside said vehicle. An adaptive display and interface mayalso be mapped to one or more surfaces on the outside of one or morevehicle such as doors, windows, ceiling panels and wheels or mappedacross the entire external surface of a vehicle and content may beadapted or displayed across one or more stationary or moving vehiclesbased on the spatial position, location and orientation of one or morevehicles to one another, to stationary objects and surfaces such as abus stop, billboard, building or store front display or to one or moreviewing parties.

In some embodiments, the AGUI system supports a variety of inputdevices. These may be traditional devices such as a keyboard, mouse,touchscreen, knobs, and buttons. A user may also interact with a hardsurface or object such as a baseball bat or tennis racket that mayprovide sensor input (ball hit position, speed, velocity, acceleration)to run a game or a teaching tool that can make use of this data. In asituation where multiple surfaces are available, one may be designatedas an active surface (such as a bat) while another may be used forinformational output (such as performance information). The baseball bator tennis racket may be equipped with sensors and report sensor data tothe main AGUI device. The sensors can report information such asposition, velocity, acceleration, and impact force. Alternatively, thebaseball bat or tennis racket may be passive, without sensors, and themain AGUI device will detect the position, velocity, and acceleration ofthe bat or racket using its built-in sensors.

Embodiments of the AGUI system incorporate a variety of sensors toprovide input to the system. Sensors may be integrated into any of theAGUI components such as projectors, LCD, LED, OLED, or other screens anddisplay types, computing and interfacing devices, and surfaces. They mayalso be connected via a wired or wireless interface. Sensors may also beon or within a user's body and may include medical devices such as apacemaker. Clothing or other flexible material may contain embeddedsensors including sensors that indicate the location and facing ofimaging elements embedded in the material.

One embodiment of the AGUI design comprises a wrist device that is wornby a user. The wrist device is an electronic computing device runningapplications, an operating system, and a virtual storage system. Thedevice comprises a number of components such as a CPU, volatile andnon-volatile memory, an integrated projector, a light or opticalimaging, depth map and sensor mapping system, an onboard OLED display,and a button keypad. Sensors are included, such as a GPS, a gyroscope,an accelerometer, an ambient light detector, user heart rate andtemperature monitor, and surface response sensor. For communications,the device may include a manifold connector and wireless device sensors.

When turned on, the onboard OLED display may show a startup splashscreen, welcoming the user. Buttons on the keypad are mapped to a set ofapplications that allow access to common functions, including but notlimited to email, web browsing, clock, calendar, contacts, and acalculator.

One typical operating mode of the device has the device attached to theuser's wrist, with the projector placed so that its display is directedonto the user's palm and fingers. The built-in mapping system identifiesand quantifies this scenario. The projector projects a light pattern onthe user's hand and senses it with the light imaging module. Raw mappingdata consists of signal relationships between sent and returned signals.This mapping data is submitted to analysis that generates extent datafor identified zones in the target surface (areas of a hand) that arecandidates for display. Areas within the projector's light path satisfyan initial criterion for a display zone. If such a candidate areafurther satisfies setup position and size requirements, it qualifies asa display zone. Typically, the mapping system and the intelligencesurrounding it identify zones corresponding to a central palm area andfinger areas that extend out from the palm area.

The data for a single zone in this scenario contains a variety ofgeometric information such as the critical distances between theprojector and the extremities of zone that define the zone's positionand shape in detail and projection angles between the axis ofprojector's output stream and an averaged normal vector to the zone.

In order to more readily explain different approaches of the presenttechnology, some embodiments will now be described in more detail withreference to the accompanying drawings.

FIG. 1 illustrates an embodiment of an AGUI system that comprises anAGUI device 201 mounted on the ceiling or wall in a bedroom. The AGUIsystem detects when the user 1002 is lying prone on the bed 1222 andtheir location, including the location of their eyes and point of focus.It displays a clock and weather report 502 on the ceiling 1212 at aconvenient viewing angle for the prone user.

Referring to FIG. 2, when the user 1002 sits up on the bed, the AGUIsystem 201 detects this and realizes that projecting on the ceiling 1212is no longer optimal and moves and adjusts the display to be on the wall1214, near the center of the user's field of view 450. Reacting topreprogrammed situations, such as a wake-up alarm, an alarm clock, andweather report is displayed in the new vertical display position. TheAGUI 201 has also added a schedule to the display based on predefinedpreferences or based on a user voice command or a motion gesture.

Referring to FIG. 3A, when the user later stands up, the user can beseen making gestures, in this case with his hand, to control the AGUIsystem. Referring to FIGS. 3A and 3B, the user 1002 selects the“Wardrobe” button 520 displayed on the wall which instructs the AGUIsystem to project an image a piece of clothing on the bed 822. In thiscase, the AGUI system has detected the bed as a surface that may be usedas a display surface and decided that it makes the best viewing surfacefor clothing given the user's position and facing, and the surfacecharacteristics of the bed at the time of use. If the bed is unmade orthe cover of the bed makes projecting a clear image on it difficult, theAGUI system would choose an alternative surface. Similarly, if there hadbeen an AGUI aware television or monitor in the room, the AGUI systemmay choose it as a display.

FIG. 4A shows an embodiment of the present technology where the AGUIdevice 201 is positioned high on a wall in a living area of a home. TheAGUI device detects when a user 1002 enters the room and maps 306 theirbody. The device detects the user's hands and based on predetermined oruser input projects 308 a user interface on the user's hand. The userthen uses their other hand to manipulate the AGUI user interface usinggestures or by manipulating the interface projected on their hand. Inthis case, the user has positioned their palm so that the AGUI device isable to clearly project a display image on it. If this had not been thecase, the AGUI system would have chosen an alternative surface in theenvironment, such as another body part, a wall, table, floor, or othersurface.

FIG. 6A illustrates another embodiment of the present technology wherethe AGUI system consists of a wrist mounted device 101. Multiple lightprojectors 108 on the AGUI illuminate the front and back of the hand402, wrist, and forearm 404 of the user. Also referring to FIG. 6B, theAGUI system maps out the position of the body parts, detects that theuser's fingers are together, and determines that it can project a largedisplay surface 330 on the palm of the hand, including the fingers. Theprojectors 108, light sensors 110, cameras or camera array 112, and/orprojectors 115 can be located on one or more edges of the wrist mounteddevice 101 to project images on either side of the device 101. Thedevice 101 can further include infrared (IR)/light sensors 114 andhaptic feedback sensors 116.

Referring to FIG. 7A, the AGUI device 101 detects the placement andorientation of the hand and fingers and determines that the palm may beused as a larger surface, to display a map, and that the fingers can beused to display control surfaces 532. The AGUI device 101 continues tomonitor the placement and orientation of the hand and fingers 402 anddetects that the user has spread the fingers of their hand 402 requiringit to adapt the position of display control surfaces 532. The AGUIdevice determines the field of view of the user and the orientation ofeach of the five display surfaces and optimizes the projection shape,color, intensity, and other characteristics of the display so that eachof the five display areas 532, 534 are clearly viewable, withundistorted color and shape. The AGUI device continuously monitors theposition of the user's hand, detects movements and projects an optimizedimage based on the position of each surface at that point in time.

FIG. 8 illustrates the detection and use of potential surfacesidentified by an AGUI wrist mounted device 101 incorporating an internaldisplay 102. The identification of surfaces and division of the surfacesinto sub-surfaces can be made dynamically based on the programming ofthe AGUI application, user defined surfaces, by utilizing a past set ofdisplay surfaces from the last time the application was run, or anycombination of methods. In this case the AGUI device 101 is aware of itsinternal display 102 and identifies a large surface 338 on the innerforearm of the user, an additional surface 332 on the palm of the user'shand, and more surfaces 334 on the user's fingers and thumb. The AGUIsystem then decides that it may reduce the size of surface 338 to createthree control surfaces 336, and that it can use each segment of thefingers and thumb as separate surfaces, 336. Finally, in FIG. 8C, theAGUI device has chosen to display an adaptive interface using controlicons 530 on internal active display 102, two display surfaces 534, andnine more icons 530, and two menu controls 532.

FIG. 9 illustrates alternative divisions of surfaces based on the AGUIdevice's 101 internal screen 102, the user's forearm 338, back of hand,and back of fingers. The AGUI system may maps surfaces differentlydepending on the position and viewing angle of the hand and fingers anddynamically select and optimize the display as the user moves theirhand. An AGUI system may detect and select a different selection ofdisplay surfaces depending on whether the user has the palm side or backside of their hand facing them. They may also display on both sides oftheir hand simultaneously to both view and manipulate the AGUI interfacefor themselves and also for a viewer located in front of them. Displaysurfaces on the palm side of the arm and hand may be projected on by awrist mounted AGUI device. Display surfaces on the back side of the handmay be projected on by another, external AGUI device.

Referring to FIG. 10, an AGUI device 101 is projecting a display surface534, containing a map, on the user's forearm. The AGUI device is alsoprojecting menu controls 532 on the user's forearm. Furthermore, theAGUI device has identified the user's hand and finger as a pointingdevice and tracks the user's hand 420. When the AGUI device detects thatthe finger has touched the SEARCH menu control 520, it instructs therunning application program to display a sub-menu 505. The AGUI deviceis aware of the size and placement of the user's hand and that it is inthe field of view of the user and selects the user's palm and fingers todisplay the sub-menu 505 comprising a large display surface 534 andfurther menu controls 532.

Referring to FIG. 11, an AGUI system may consist of a number of AGUIdevices that are networked together. A wrist mounted AGUI device 101 maybe used together with an external AGUI device to implement a userinterface 530, 532 on a user's arm 404. Whether the AGUI system maychoose to use the wrist mounted AGUI device 101 or an external, remoteprojector 510 to project the content onto the user's arm 404. It can beobserved that since the user's hand is in the shape of a fist, the areason the palm and fingers are not available as display objects. FIG. 12shows how the AGUI system adapts when the user then opens their hand422. Now the AGUI system detects their palm and finders as potential anddecides to use them as display surfaces 534, 505. The AGUI system mayuse either the wrist mounted AGUI device 101 or a remote projector toproject any of the display surfaces independently.

FIG. 13 illustrates how the addition of a wrist mounted AGUI device 101provides and additional display screen 102 that allows the AGUI systemto display additional information.

Referring to FIG. 14, an AGUI system may also be used with a passivedisplay object 350. In this example, a rectangular slab is held by theuser. The AGUI detects the presence of the slab as well as its positionand extents. The slab may be composed of a particular material and havea texture or color that is amenable as a display surface. The slab maybe passive or may incorporate sensors and electronic components thatallow it to communicate with the AGUI system and report to the systemits location, movement parameters, and orientation. This form ofelectronic slab may contain an identifier or a set of parameters thatallow the AGUI to obtain a set of predetermined optimal displayparameters to be used when projecting on it. It may also include otherenvironmental sensors that measure the like levels, temperature andother factors that may affect the operation of the AGUI system.

FIG. 15 shows a similar embodiment of the present technology except thatthe AGUI system here is a wrist mounted device 101 that detects andprojects 360 onto the slab.

FIG. 16 illustrates a similar interface but where the slab is replacedwith a table 720, desk, or similar horizontal surface. The table may bepreregistered with the AGUI system which may be done by dynamicallydetecting a flat surface with a color and texture that makes it a goodcandidate or a display surface, or it may be chosen by the user andremembered as a preference or as having been used the last time.Gestures may be employed by the user to indicate the exact position andsize of the area of the table to be used as a display. The AGUI systemmay also be used with a passive display object and/or depth mapping &imaging of an object 350 that can project interactive multimedia content510 from remote projector, interactive tabs/menu items 532, and/orinteractive multimedia content 534.

FIG. 17 shows a similar embodiment except that the display surface is avertical surface 1214. The vertical surface may be a wall, the frontpanel of an appliance, or suitably treated transparent glass or plasticpanel. The user may use their hand 1102 to make gestures or to touch theGUI elements 520 projected onto the surface 1214 to control the system.Alternatively, vertical surface 1214 may be capacitive or resistivetouch sensitive and detect the user's touch.

FIG. 18 illustrates an embodiment of the present technology comprisingAGUI enabled wearable devices and AGUI enabled eye glass and ear phones.The user 1002 is shown wearing a jacket with an AGUI device attached orincluded in the form of a lapel pin 130. In FIG. 18C an AGUI enabled earpiece 150 is also included and is in communication with the lapel pin130. An imaging sensor and projector which may be incorporated intoeither the lapel 130 or earphone 150 and, either on their own or workingtogether, can perform the detection, processing, projection, and displayfunctions characteristic of an AGUI system. The system allows forvirtual or augmented reality applications where a virtual object 714 isprojected onto the user's hand 402 and may be manipulated in space bythe user 1002. It can be appreciated that the AGUI enabled wearabledevices can be a head mounted device.

FIG. 19 illustrates how the AGUI system can track and project imagesonto objects being manipulated by a user. In the example given, the userholds a ball 714 in their hand 1102 and the AGUI system 101 projectsinteractive multimedia content, in this case an image of a globe 534 onthe ball. The AGUI device 101 can also update the image to show adisplay surface 534 and control icons 530 that may be used to implementa game or other computer application.

The image projected may also when the user holds the ball 714 so that itis detectable and is viewable by the AGUI system 101, the globe isprojected on the ball. Note that in an AGUI system with multipleprojectors, as long as one projector has a line of sight to the ball714, the globe may be projected on it. Alternatively, the ball itselfmay be an active device that can connect to the AGUI system usingwireless networking protocols and have the ability to project the globeimage on itself. The ball may also include sensors to communicate,position, velocity, and orientation in three dimensions as a user throwsit. Portions of the ball may be obscured by the hand and the projectorin the wrist mounted AGUI device may be unable to project onto it. TheAGUI device will detect what portion of the ball is visible by itsprojector and project only onto the visible portion. Though a ball 714is shown, any other tool, sports, or game device may also be used in asimilar manner.

FIG. 20 shows an embodiment of an AGUI system comprising an AGUI deviceincorporated into eye glasses, headset, or other head mounted device150. The arms or body of the eye glasses contain both sensors, includingcameras 160, to sense the environment, processing and networkinghardware, and projectors 156 to project a display on one or both lenses156. Eye facing cameras 158 may be used for eye tracking and the resultscan be used to optimize the heads-up display or to provide contextsensitive information based on that the wearer is viewing. The resultingheads-up display may include display surfaces 154 as well as controlinterfaces 540.

As best illustrated in FIG. 20, in some embodiments, the AGUI systemControl buttons and/or speakers can be included on one or both of thearms.

FIG. 21 shows how an AGUI system comprising a virtual reality (VR) oraugmented reality (AR) headset 168 can be used for the visualization andmanipulating of virtual objects 802 using multipoint control in space.In these embodiments, the user can use their hands to manipulate andcontrol a virtual 2D or 3D object in space. FIG. 21B shows a user 1002using an augmented reality headset 170 with an augmented reality visor172 that is part of an AGUI system. An augmented user interface 545 andcontrols is viewed in the user's field of view is augmented and may becontrolled using eye tracking, hand 420 gestures, and other techniquesas known in the art.

FIG. 22 illustrates how the AGUI system may adapt the layout of theinformation it is displaying based on the shape of a display surface.When utilizing a square 704 or rectangular display the display elements,the rectangular map view 534 and the labelled buttons 532 are laid outwith the map on top and the buttons below in a grid like arrangementsuited for a rectangular display. In the circle or oval display 720, themap 534 is now positioned in the center with the labelled buttons 532placed around them in a circle, making better use of the availablespace. During use, the AGUI system may change the displays used or theshape of the display and may dynamically change the layout of displayedinformation to make better use of the display shape or in accordancewith program or user preferences.

FIG. 23 illustrates similar concepts where the AGUI system implements aresponsive display that adapts the display layout based on the size ofscreen available when using a laptop computer 704 or tablet versus acellular phone 706. Also illustrated is the dynamic adjustment of layoutoptimized for different shaped small screened devices such as wristwatches 708.

FIG. 24A illustrates a garment 601 incorporating display fabricsaccording to another embodiment of the present technology. The garment601 shown includes an active display area 515 including display elements534 and control elements 520. The display elements may be a matrix ofilluminated threads or individual embedded picture elements and may useorganic display elements based on OLED technology. Each picture elementmay also comprise a position sensor that is used by the AGUI system todetermine the optimum display parameters from the point of view of theviewer. Display elements that are hidden by folds of the cloth may beturned off. Indirectly viewed elements may be displayed more brightlyand directly viewable elements may be displayed less brightly to producethe overall effect of a uniform brightness display. As the cloth moves,the AGUI system may detect this and adjust the brightness, color, orother display parameter to dynamically produce a uniform display as thecloth moves. The user 1002 may select a control interface, such as amenu item 520 which can cause the display 515 to update to initiate aprogrammed event. FIG. 24B illustrates and embodiment where the user1002 has an AGUI enabled jacket 601 where the user is able to view animage 515 in the fabric and control the AGUI jacket itself by selectingfeatures on the image 515. The AGUI jacket may detect user's selectionusing resistive or capacitive embedded sensors in the fabric or a lapel,head mounted or externally mounted AGUI system, networked to the AGUIjacket, may detect the location of the user's hand 420 making controlgesture or movements.

FIG. 25 shows additional examples of an AGUI system using a garmentincorporating an active display. Different which may implement avirtually reality sound system including displays on the front, back,and side of the garment. Display elements such as QR codes 832 can alsobe displayed for advertising applications.

FIG. 26 illustrates how multiple AGUI enabled garments, in this case ajacket 602 and pants 603, may be networked and work together as a singleAGUI system. As they are work together and will be used in closeproximity to each other, the two garments may be networked using shortrange wireless protocols such as Bluetooth, as well as other networkingprotocols. When used together, the images displayed may be independentof each other but may also span the two garments and act as a singledisplay.

FIG. 27 illustrates the operation of an AGUI system as a user movesthrough an AGUI enabled environment 1202. The AGUI system comprisesseveral AGUI devices positioned throughout the house and may includeadditional AGUI devices on the user. The AGUI system may also act as ahome hub to control appliances, HVAC system, doors, locks, curtains,blinds, lights, security systems, and any other IoT (Internet-of-Things)enabled devices. There may also be active AGUI enabled displays such aslaptops, cellphones, tablets, television, etc. AGUI devices may also beincorporated into active displays. The networked AGUI system maps outthe environment in 3D 1202 including the user. As a user 1002 movesthroughout the environment the AGUI system detects their location, fieldof view, velocity, acceleration, etc. and determines what AGUI displaysare viewable by the user. The AGUI system dynamically determines thebest displays to use as the user moves about the environment. The usermay use gestures to indicate display preferences, including the size andshape, and also use gestures and traditional computer input methods tointeract with the system.

As best illustrated in FIG. 27, in some embodiments, the environment1002 can be, but not limited to, a building including an entry way 1210,a bedroom 1220, a kitchen 1230, a living room 1249 and/or a dining room1250.

FIG. 28A illustrates an embodiment of the present technology where anAGUI system comprises an AGUI device 201 located on a table 1252 betweentwo users 1002, 1020 that allows the two users to collaborate with thesame user interface 534. The AGUI system detects the location and fieldof view of the two users and also detects body parts of the users thatcan be used to indicate gestures to control the AGUI system. The AGUIsystem has queried the surfaces in the environment and selected thedesktop between the two users as the best display surface. The AGUIsystem can use a single or multiple AGUI devices to present displaysoptimized for either or both of the two users. Both users cansimultaneously use the system or it may be configured that one user mayonly view content while the other may both view and control content.FIG. 28B illustrates how the same AGUI system may also support twoindependent user interfaces to the two users. The AGUI device may beconfigured to select a separate surface 1402, 1404 for each user togenerate independent user interfaces 536 538 for each user. Asillustrated in FIG. 28C, the AGUI system maps out physical objects onthe display surface 1252 and removes areas that may not be used wherethe physical objects 730 prevent it. The remaining area 320 may then beused as is, or subdivided into a number of display surfaces. Similarly,for active surfaces, portions of an active display that are obscuredfrom the user's view can be omitted from the surface and the remainingsurface may be used as a single or multiple surfaces. When physicalobstacles are moved within a display surface, introduced or removed froma display surface, the interface adapts in real time. Portions of adisplay surface may be moved to active displays, such as a laptop or TV,or be moved to a new surface. The display content may also bereorganized or resized to fit the updated available surface.

FIG. 29 illustrates how an AGUI system can be used together withhousehold appliances 360 to enhance their usefulness. The systemcomprises an AGUI device 201 in proximity to the appliance, which may beAGUI enabled or not. It the appliance is not AGUI enabled the AGUIsimply uses a surface of the appliance as an AGUI surface to project adisplay on. Preferably the content projected on the appliance surface isrelated to the use or content of the appliance which would be thecontents of a refrigerator or settings for baking with an oven. Theinterface projected onto each appliance could allow a user to inputsettings to for the appliance as well as monitor the contents of theappliance. This may include the amount of remaining food 850 in therefrigerator or a control icon to initiate online purchases 852. Anappliance may also be AGUI enabled which would allow it display contenton an active display surface embedded or mounted on the face of theappliance. The AGUI enabled appliance could also include internal andexternal sensors that would provide up to date information on thecontents and operation of the appliance. FIG. 29B illustrates how AGUIinterfaces may allow a user to not only control appliances such as amicrowave oven 1236 or an oven 1235 but also indicate the status of thecooking. As shown in FIG. 29C, two AGUI devices 201 may be networkedtogether to implement a more complete user interface where if the lineof sight from one device to a surface in use 360 is blocked for one AGUIdevice, the other can project an optimized image. As the user moves inthe environment, a network of AGUI devices can collaborate to ensurethat a surface has complete display content no matter how the user movesor where they are located.

FIG. 30 illustrates how the AGUI system comprising a single or aplurality of AGUI device 201 may be able to implement virtual realityapplications by projecting images onto the floor and walls of a room.The AGUI device may be arranged on a track 224 to give adequate of theroom. Couch and table 810 may be displayed in an augmented reality viewto allow a user to view a room with a variety of in different sizes andshapes.

FIG. 31 illustrates how an AGUI system of devices 201 can be used tocreate an immersive, augmented reality environment for activities suchas golf, tennis, hockey, cycling, skiing, and other sports oractivities. The user's golf club may also be AGUI enabled with sensorsto track the location, velocity, and acceleration of the golf club as itis being swung. Similar embodiments exist for other sports applicationsand applications that would benefit from an immersed environment orvirtual reality. The TV 720, an active display may be used as a separatedisplay as in FIG. 31A or may be included in the other augmented realityelements 866 as shown in FIG. 31B.

FIG. 32 illustrates a further class of embodiments that center aroundvehicular applications. A vehicle 900 incorporates a number of AGUIsensors 950 and detects when a user 1002 approaches the vehicle. TheAGUI system may use a number of biometric factors to verify the identityof the user. The vehicle may then take a number of actions based on theuser identified. This may include unlocking the vehicle, displaying akeypad on the door, or sending a message. The vehicle may be coveredwith an active surface such as an OLED or an active paint that allowsfor the surface or a portion of the surface of the vehicle to become anactive display. Windows 952, 954 may have an embedded or surface, LED,LCD, OLED, a projection display, conductive paint or other coating ordisplay system or surface so as to appear opaque unless the owner isdetected to be approaching or close by, then becoming transparent.Images may be mapped and displayed on the surface of the vehicle thatmay change depending on a number of factors including the location ofthe vehicle, the presence of other people or objects, or whether thevehicle is moving or not. FIGS. 32B and C shows how a single image maybe mapped and displayed over multiple panels and display surfaces ordifferent images may be displayed on each side, section or display panelsuch as a door, window, wheel, hood or roof of the vehicle and how thismay be dependent on who is viewing each side. FIG. 32D illustrates how akeypad 504 allowing entry to the vehicle may be displayed on the vehicleand how a video conference 552 link may be initiated with a third partyto control access to the vehicle. Remote access control can be used toimplement car rental or car sharing services.

As best illustrated in FIG. 32, in some embodiments, the AGUI systemincorporated with the vehicle 900 can perform depth mapping 306 using astructured light imaging system, and depth and color mapping and 308imaging using a camera array.

FIG. 33 illustrates the use of AGUI systems in applications usingvehicular media broadcasting. Vehicles 900 in motion are AGUI enabledwith display surfaces on panels 956, windows, wheels and/or hubcaps 845and on their roofs or roof advertisement panel 960 and on their hood,doors and other Panels 956 which may work independently or together toprovide a spanned image while hubcaps 845 and roofs 960 provideindependent displays. Images on rotating surfaces such as hubcaps 845can be rotated in real time so as to present a stable image from thepoint of view of an external viewer. Buildings 940 can also be used asactive AGUI displays or as passive projected display surfaces.Similarly, other surfaces such as the road or sidewalk 845 may also beused as active or passive display surfaces. Since the various AGUIdevices are in communication with each other they can coordinate theirdisplays. For example, the sports or themed displays in the figure mayonly be displayed when a vehicle 900 is in view of buildings 942. Panels942 may display specific content when one or more viewers are detectedor may display interactive content when one or more vehicle displays aredetected a system may only illuminate or display content when the systemdetected that there was a viewer. Similarly, the external AGUI displayswould only be illuminated when being viewed by and external viewer andwould display different themes depending on the vehicle's location.

As best illustrated in FIGS. 33A-33C, in some embodiments, additionaldisplay elements can include a virtual footballer 830, and/or a virtualfootballer's mask-helmet 831.

FIG. 34 illustrates the use of AGUI systems in application with externalviewers 1002, stationary AGUI devices 930, and AGUI devices in motion920. As the bus approaches the bus stop, they begins to communicate witheach other to dynamically form an AGUI system. A viewer 1002 is inproximity to a bus stop 930 which may initially be presenting a blankdisplay or a variety of advertisements or other messages. When bus 920arrives in proximity to the bus stop 930 the AGUI system detects thatthe viewer is present and proceeds to present a coordinated displaydistributed over the bus and bus stop surfaces. This may include rich,interactive content such as a contest. Here the bus display 845 promptsthe viewer to take a virtual swing at a ball thrown by display element834. The viewer 1002 then makes a swinging motion that is detected bythe AGUI system in order to enter the contest. Content relevant to thecontest 838 is also displayed on the bus stop 930. The displayedinteractive graphic and multimedia content may also be dynamicallyupdated, positioned and/or oriented in relation to one or more viewerssuch as the direction of the pitcher on the bus and the catcher on thebus stop in relation to the viewer. In the case of a 3D image and/orinteractive multimedia content displayed on a the bus, such as thepitcher displayed on the bus and the catcher displayed on the bus stopmay physically turn towards the viewer as the AGUI system dynamicallyadapts the displayed content to one or more viewers and display surfacesbased on the relational position, location, orientation, viewing angleand/or point of focus of one or more viewers to one or more displaysurfaces. In the case of multiple viewers the AGUI system may select anoptimum viewing angle and surface for viewing by all identified viewers.

FIG. 35 illustrates a vehicle mounted AGUI system comprises an AGUIdevice 901 mounted on the ceiling of the vehicle where it can map outthe interior of the vehicle including the location of the driver 1002and any passengers 1020. The front window may be uses to project aheads-up display for the driver. A side window may be used as a displayfor the passenger. The driver's display would be restricted by safetyand agronomical considerations and in many embodiments, would be definedby the vehicle manufacturer. Passenger displays would offer moreflexibility and allow passengers to see a wide variety of information.

The AGUI device 901, as best illustrated in FIG. 35, in someembodiments, can include interconnectable modules, and can feature anoval, spherical, curve or other geometrical configuration.

FIG. 36 shows an alternative implementation where the AGUI device iscontained in the driver's eye glasses or a head mounted AGUI device. Inthis way the AGUI system would follow the driver about and could offersecurity features such as the vehicle would not start of move without anexchange of information with the driver's AGUI device.

As best illustrated in FIG. 36, in some embodiments, a vehicle head-updisplay/Interface 550 can be projected onto, for example, a windshield952 or window of the vehicle. It can be appreciated that the AGUI devicecan be a heads-up projection interfacing system configured to displayonto the windshield 952 or window of the vehicle.

FIG. 37 illustrates how AGUI devices may also be included on thedashboard or together with the rear-view mirror. An AGUI system may usea single AGUI device or multiple, networked devices. This could includea dashboard device to project the drivers display, a ceiling mounteddevice to implement displays for passengers, and a device on thedriver's person to provide security and authentication services.

FIG. 38 illustrates the use of an AGUI system in an office environment.An AGUI device 101 is placed on a worker's desk to display userinterfaces on the user's desk 504 and cubicle walls 534. The user willhave the ability to use gestures or traditional keyboard and mouse inputdevices to select surfaces as display surfaces, position them, selecttheir size and shape, and decide the content of each display.

A webcam 718, as best illustrated in FIG. 38, in some embodiments, canbe in communication with the AGUI device 201 to provide additionalcamera input. Virtual 3D objects 802 can be projected onto the desk 504and/or walls 534.

FIG. 39 illustrates the use of an AGUI system in a medical environment.An overhead lighting system incorporates an AGUI system 1274 which canproject informational or soothing images for a patient 1026 pre or postoperation. Video conferencing 536 may also be conducted with family,friends, or medical care givers.

FIG. 40 illustrates how an AGUI device 201 may aid during an operation.The AGUI system has identified a surface on the patient's cover and isable to display the patient's vital signs 560 and other information in aconvenient location to be viewed. In some applications, medical AGUIdevices, internal and external to the patient, may be used to gatherpatient information; internal imaging, blood analysis, monitoring ofvital signs, and other. The AGUI system may then process and prioritizethe information and display a customized user interface to the membersof the surgical team. Artificial intelligence may be used to detectanomalies and inform the medical staff at an early stage in theprocedure.

FIG. 41 illustrates the use of a flying drone as an AGUI device. FIG. 42shows how an AGUI device 1002 may be used to perform delivery of goods.Here the flying AGUI drone detects or communicates with the packagerecipient. This may be done by the recipient 1002 wearing an AGUIenabled garment or wearable device, or by the being in possession of anAGUI dongle or similar device. Once the AGUI drone has authenticated theuser, the package may then be delivered. FIG. 43 illustrates how in someembodiments, the AGUI drone may display a user interface 840 withdirections relevant to the package or in order to obtain a signature orpayment authorization from the user.

FIG. 41 illustrates the use of a flying drone 750 as an AGUI device. Itcan be appreciated that the AGUI device can be integrated with orattachable to the drone 750, and can include an aerodynamic body.

In some embodiments, the AGUI drone 720 can lower the package by way ofmotorized and/or retractable cables, as best illustrated in FIG. 42.

FIG. 44 illustrates the use of a rolling AGUI device such as the robotillustrated 722.

FIG. 44 illustrates the use of a rolling AGUI device such as the robotillustrated 722. The AGUI device can be attached to or replace a headunit of the robot. It can be appreciated that in some embodiments theAGUI device can be attached to or replace any part of the robot.

FIG. 45 illustrates how a number of AGUI device may be networkedtogether in a complex environment with a number of users in motion.

FIG. 46 illustrates how an AGUI device 201 may be constructed in amodular way. A track 224 is mounted and supplies power, and optionally,networking wiring. A track mount 225 for the device is coupled to amicrophone and speaker array 206. The microphone and speaker module 206is couple to a module 216 with light sensors and camera arrays. Finally,a light module 220 is attached so that the entire AGUI device 201 hasthe look of a typical track lighting module. FIG. 46 illustrates the useof a halogen or LED bulb 223 in the alternate light module 222.

FIG. 47 shows how the module components may be interchanged. Here themicrophone and speaker module 206 has been replaced by a similar modulethat screws into a standard light fixture socket. An alternate lightmodule 222 is also used to provide a choice to the user. FIG. 48illustrates how the light fixture adapter 214 may be removable andreplaceable with other adapters to fit other types of fixtures such asbayonet type light fixtures and light fixtures of various diameters.FIG. 49 illustrates how a variety of base units 240 may be useddepending on the application.

In the some embodiments of any one of the AGUI systems described herein,if the AGUI system determines that a projectable surface is detected,then the surface is mapped and characterized based on the mapping. Afterwhich, the AGUI system will select an appropriate projection scenariobased on the surface characterization. Following this, the AGUI systemwill create a display based on the scenario, the current surface mappingand current desired display.

If the AGUI system determines that a projectable surface is notdetected, but detects a surface that can create display and that thesystem can communicate with, then an external display scenario iscreated. Following this, the AGUI system will create a display based onthe external display scenario. It can be appreciated that the AGUIsystem can create a display for both the projectable surface and theexternal display simultaneously.

It can further be appreciated that when two or more buses or vehiclesare in close proximity the AGUI system can share information, displaycontent and/or an interface between them, essentially creating a mobilenetwork.

The present technology can include a controller system includingcomputing hardware components and programming configured or configurableto perform operations associated with functions of the presenttechnology device. Sensor subsystems including one or more sensorscapable of detecting user and display surfaces can be utilized with orin the present technology. Display subsystems including one or moreprojectors capable of projecting image/data onto display surfaces and/orcontrolling peripheral display device can be utilized with or in thepresent technology.

The present technology is capable of continuously mapping itssurrounding environment and can determine userlocation/orientation/field of view and/or user body partlocation/orientation, and/or optimal display surface(s).

The present technology is capable of continuously monitoring user and/oruser body part movements for controlling the display subsystem and fordetermining user interface inputs associated with the displayedimage/data.

One or more sensor arrays can be utilized in associated with mapping theenvironment and detecting user control interface.

In some embodiments of the present technology, each individual projectorcan be moveable to change its direction of projection, and/or the entiredevice can be moveable to change the direction of all the projectors.

The AGUI system of the present technology can utilize one or moreMulti-Directional Smart Projector (MDSP) systems 201 that can include amounting fixture configured or configurable to mount, support or attachto a surface or object, thereby allowing it to project images and senseconditions within an environment. The MDSP system 201 can be a compactdevice that embodies the general AGUI operations. The MDSP system 201 iscapable of working and communicating with other MDSP systems or AGUIdevices that might not be an MDSP system.

Some embodiments of the present technology can include a plenopticsystem. As best illustrated in FIG. 49D, which is a plenoptic systemaccording to one embodiment, the plenotic system can comprise anobjective lens 72 that forms an image on a microlens array 74 and a CMOSsensor 76 that is placed in the focal plane of the microlens array 74.FIG. 49D is a diagrammatic representation of a plenoptic image systemaccording to an embodiment that can be utilized in any one or moreembodiments of the present technology AGUI system or device such as butnot limited to the AGUI device shown in FIGS. 49A-49C, any one of theAGUI devices of FIGS. FIGS. 46A-46B, 47A-47C, 48A-48B or otherembodiment(s) of the AGUI device.

An example of such an optical system can be found in the Apple iPhone 5ssmart phone. In this example, the CMOS sensor 76 measures 4.8×3.6 mmwith a pixel size of 1.5 microns. The array consists of 7,680,000pixels, and is termed an ‘8 megapixel’ sensor. It can be appreciatedthat other optical systems can be utilized with the present technologyin replace of a smartphone camera or the described 8 megapixel sensor.Specific cameras or CMOS sensors with predetermined characteristics suchas, but not limited to, resolution, megapixels, field of view, zoommagnitudes, etc. can be incorporated or utilized with the presenttechnology.

The focal lens of the camera lens is computed from the size of the CMOSarray 76 and the angular field of view of the camera. In the smallersensor dimension, the field of view is 48 degrees. The focal length ofthe lens 72 can be given by equation 1.

$\begin{matrix}{f_{obj} = {\frac{3.6}{2 \cdot {\tan \left( \frac{48{^\circ}}{2} \right)}} = {4.04\mspace{14mu} {mm}}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In following this example, a focal length (f_(microlens)) of themicrolens 74 can be 0.5 mm and pitch (distance between lens centers) be0.125 mm. The microlens array 74 is the same size as the CMOS sensor 76.This permits the creation of a 38×28 element microlens array.

The f-number is defined as focal length divided by the aperturediameter. The f-number of the objective lens 72 is the same as thef-number of each lens forming the microlens array 74.

The microlenses 74 have an f-number=f/4. For exemplary purposes, cellphone camera lenses typically have an f-number=f/2.2. Thus the cellphone camera lens is stopped to a 1 mm diameter aperture. If a longerfocal length lens is used, the desired aperture diameter is larger.

This is a possibility for the MDSP system 201 because the physicalthickness limitation of a cell phone does not apply to the MDSP system.However, the objective lens focal length can only be slightly increased,since as the focal length increases, the field of view decreases. Toachieve a field of view similar to a smart phone and using a smart phonesensor, the focal length of the objective lens 72 needs to be similar tothat used in a smart phone.

Summing the focal lengths of the objective lens 72 and the microlens 74,and allowing for lens and component thicknesses, the optical systembetween 5 mm and 6 mm thick with area equal to the CMOS sensor 76 andits mounting. If a longer focal length objective lens is used, the totalthickness is greater.

To use such a plenoptic system, the CMOS camera can be connected to animage processing system containing a graphics processing unit (GPU).

The above design allows a dense sampling of the angular component of thelight field captured by the plenoptic system due to the high count ofpixels included in the sensor image associated with each microlens. Butthe spatial resolution, which is set by the number of microlenses in thearray, is low. Thus, reconstructed images may have low spatialresolution.

Referring now to FIG. 49E, an alternative exemplary plenoptic camerasystem can be utilized with the present technology. FIG. 49E is adiagrammatic representation of a plenoptic image system according to anembodiment that can be utilized in any one or more embodiments of thepresent technology AGUI system or device such as but not limited to theAGUI device shown in FIGS. 49A-49C, any one of the AGUI devices of FIGS.47A-47C, FIGS. 48A-48B or other embodiment(s) of the AGUI device. Thisalternative exemplary plenoptic camera system can comprise of theobjective lens 72 that forms an image on the microlens array 74 and theCMOS sensor 76 that is placed in the focal plane of the microlens array74. However, the focal plane 78 of the objective lens 72 is located apredetermined distance “a” from the microlens 74, with the CMOS sensor76 being located a predetermined distance “b” from the microlens 74.This alternative exemplary plenoptic camera design is configured orconfigurable to balance angular and spatial resolution while usingsubstantially similar components found in the optical system of FIG.49D.

The CMOS sensor and the microlens array are placed so that the CMOSsensor is imaged by the microlenses in the focal plane of the objectivelens. Object distance=“a”; image distance=“b”. The overall system lengthis larger than the first design but this can be accommodated in a MDSP.The values of “a” and “b” are adjusted to allow the spatial resolutionof reconstructed images to be increased while decreasing the density ofangular sampling.

The MDSP system 201 can include a micro laser projector. An exemplarymicro laser projector utilizable in the present technology can be anoptical system that projects images onto a surface using a combinedlaser beam consisting of red, green and blue beams. Full color imagescan be formed on the surface utilizing such a micro laser projector.

In the exemplary, embodiments of the present technology can utilize ascanner consisting of a programmable Micro Electro Mechanical System(MEMS) mirror. Such an MEMS mirror can be utilized with the opticalsystem or other components of the present technology. The MEMS mirror istilted under program control to direct the combined laser beam tovarious locations of the display area thus scanning the beam over thedisplay area and so doing creating the display. As the MEMS mirror scansthe combined laser beam, at each location the intensity of each of the 3lasers making the combined beam is set by the program thus producing thedesired color and brightness at that location. Given the limitedexpansion of laser beams as they propagate, the image is in focus overan extended depth of field.

An example of such a projection system is the MicroVision PicoP system.The physical dimensions are: thickness=6 mm; width=36.3 mm; depth=53.4mm. It consists of two parts, the optical portion (IPM) and theelectronic control portion (EPM). Both the IPM and the EPM have the samewidth, but the depth of the EPM is somewhat less than that of the IPM.

With IPM and EPM separated, both can fit into an enclosure of the MDSPsystem. Several such units facing in different directions may beincluded in a single MDSP system 201, thereby providing a wide angularrange of coverage, including 360 degree coverage.

In some embodiments of the present technology, a camera and projectorused as a pair can be utilized. The projected field of the PicoP systemcan be 45 degrees. This is comparable to the exemplary plenoptic camerasystem described above. The camera can image the entire projector image.This is true at all distances from these two systems if they are placedtogether and have the same central direction of view.

The camera system image can provide the stimulus to modify the projectedimage so that the intensity and contrast of the viewed image is thatdesired by the AGUI system. Input image data to the projector system canconsist of three arrays of intensity values, one for each color. Whenthe observed intensity (camera values) for each pixel and each color arenot the values desired, the AGUI system can modify the image data sentto the projector system to correct for this difference.

There may be a limit to this correction, in that each laser has amaximum intensity. But such a system is able to compensate forvariations in back scattered light due to changes in the surface onwhich the image is projected.

Arrangement of the camera/projector pairs within the MDSP system 201 canbe utilized. A span of 360 degrees of projection and viewing the MDSPsystem 201 can be accomplished with eight projector/camera pairs 202,each pair covering just over 45 degrees of field of view, as bestillustrated in FIGS. 49D and 49E. Using two layers of eight pairs each,the useful covered vertical space ranges from substantially 20 degreesabove horizontal to 65 degrees below horizontal. This is achieved bytipping one set of eight pairs downward 45 degrees with respect to theset that projects nominally horizontally. It can be appreciated that anynumber of camera/projector pairs that cover a predetermine field of viewrange can be utilized to result in 360 degrees of projection andviewing.

The intensity of the laser beams is such that there is intense backreflection from any simple transparent exit window in the MDSP system201. It is beneficially that such reflections should not fall on anyplenoptic camera sensor in the MDSP system 201. A solution to this laserback reflection problem can be to place the objective lens 72 of theplenoptic camera system on the outer surface of the MDSP system 201, inthe same plane as the exit window of the micro laser projector, andsurround the rest of the camera with an outer opaque skin. The windowreflections would come back into the MDSP system 201, but can neverreach the camera sensor.

The MDSP system 201 can include electronic for operation of the opticaland/or camera systems. The MDSP system 201 can be a self-contained unitcomprising the following: electronics for image processing; electronicsfor communication in a networked AGUI system; power to operate allcomponents.

Each plenoptic camera system can include a GPU to process the datacollected from its CMOS sensor. This creates focused images anddistances to the focal plane. This camera electronic package cancommunicate with an AGUI network controller to provide it withinformation including distance data for spatial mapping. The cameraelectronics can also communicate with the EPM of the projector to sendmodifications to the image data that the AGUI network controller sendsto the EPM so that correct contrast is obtained for viewers.

Several criteria can be used to determine the location in space of aMDSP system 201 and the direction of view of its cameras and projectors.When there is more than one MDSP system 201 in a space, the AGUIcontroller requires the spatial location of each MDSP system 201 inorder to generate a composite map of the environment and/or display areaand objects within it. The AGUI controller requires location anddistance information from the field of view of each MDSP system 201 andlocation and view direction of each MDSP system 201. With this data, theAGUI system can direct the projectors of each MDSP system 201 to formimages in desired locations with desired content. The MDSP system 201acquires or computes the sensed location and direction data similar tothat available in smart phones.

It can be appreciated that the AGUI system and/or MDSP system 201 canutilize other camera systems. The MDSP system 201 can use camera systemsother than the plenoptic system to provide 3D locations of objects inthe camera system field of view.

One such system that can be utilized is a ‘time-of flight’ camera. Suchsystems offer the same capabilities as plenoptic camera systems. Howeverthese systems can be more complex in terms of physical components andelectronic control.

Two cell-phone type cameras can be used as a pair. The optical axes ofthe paired cell-phone cameras are horizontally separated, for examplelike the separation of human or animal eyes. This provides stereo visionor imaging. In the MDSP system 201, such a camera stereo pair is in turnpaired with the projector system described above. The two camera imagesare input to a GPU that correlates the images and extracts the depthpositions of objects using parallax.

Some embodiments of the AGUI system can utilize multiple MDSP systems201 within a given display space. These MDSP systems can be incommunication with each other and/or other peripheral devices orsystems.

Use of several MDSP systems 201 in a given space can provide advantagesover a single unit. The cameras of the MDSP systems 201 can image oneanother and image common items within the space or environment. Suchimages can be communicated to the AGUI system controller. The AGUIsystem controller can use the direction of view of each MDSP system 201as it views a neighbor and uses correlation of image details, and thedirection of view of each MDSP system 201 to common objects to locatethe MDSPs with respect to one another and to objects in the space orenvironment. Given this redundancy, location and direction data of allMDSP system 201 in the overall AGUI system can still be constructed.Such situations occur in spaces or environments where GPS signals arenot sufficiently strong for the location and direction sensingelectronics to work well.

Additionally, projection can still be accomplished if the projector inuse is temporarily blocked by an object in the projector path. AnotherMDSP projector with an overlapping projection field can be assigned totake over creating the display.

Use of multiple MDSP systems 201 allows projection coverage of a largerdisplay area than one projector can access, without distortion ofimages. An individual MDSP system 201 with projector and camera systemscarefully arranged can project images that completely cover the area orenvironment surrounding it. However, if a planar display area is largerthan the field of a single projector whose direction of view isessentially perpendicular to the display area, the other projectors inthe MDSP will project at a large angle to the surface, thus distortingthe images they form due this disadvantaged angle. Using additional MDSPsystems 201 whose projection fields are adjacent to the field of thefirst projector but with their direction of view essentiallyperpendicular to the display area, an undistorted image field iscreated. Such combined projected images are coordinated by the AGUIsystem controller. By using two or more MDSP systems 201, the distortionproblem may be removed without resorting to image warping.

When a large display is constructed with multiple MDSP systems 201,projected data can be scrolled from one area of the large display toanother area. With conventional technology such multiple display usagecan be realized by arranging two or more physical displays in closeproximity. In the MDSP system 201, the location of the display area issmoothly reorganized when the AGUI controller perceives that viewershave moved. The projected data may be moved to adjust to the viewers'changed locations.

The placement of multiple MDSP systems 201 in a space or environment canbe utilized for optimal performance. For optimal performance, multipleMDSP systems 201 may be placed in the space or environment in anorganized fashion. The field of view of each individual MDSP system canbe considered geometrically. The available display area is tiled by thefields of view of the multiple MDSPs. The MDSPs are placed so that theavailable display area is covered by the projectors and cameras of aMDSP without excessive multiple coverage of sub-areas. Some smalloverlap is desirable to ensure continuity as display data is scrolled ormoved from one projector to another.

The fields of view of the MDSP projectors and cameras are consideredtogether to form the composite field of view of the MDSP. Given a fieldof view expressed as a solid angle Ω, and the distance to the displayarea d, the coverage area is given by A=Ω×d.

Where the MDSPs are placed in an elevated position such as, but notlimited to on a wall or ceiling, for example as illustrated in FIGS.1-3B and 30A-31B, there is a maximum distance dmax from the MDSP to apotential display area. The field of view of the MDSP is essentiallysquare, so the solid angle is given by σ=√Ω. Maximum displaywidth=σ×dmax. If the display projected by adjacent MDSPs is notoverlapped, the MDSPs can be placed at a pitch given by this width. Toallow some overlap of display areas at the maximum distance, a slightlysmall separation distance between MDSPs is chosen.

Placement of MDSPs for most effective use is a matter of deciding on themaximum distance in which a group of MDSPs will have to display images,and then spacing the MDSPs using the considerations described above.

The MDSPs that are placed on the walls or on the ceiling near a wallmight provide inefficient operation. A more efficient way to place MDSPsin such a space is to mount them centrally on the wall or centrally onthe ceiling. Then an individual MDSP creates a display in the full spaceor environment surrounding it. A single MDSP creates a display on wallsand surfaces that are perpendicular to one another if it is placed sothat one of the fields of view of a projector/camera system covers thearea.

As best illustrated in FIGS. 46A-49C, the MDSP system 201 can be asingle integrated device or can include multiple modules such as, butnot limited to, a microphone and speaker module 206, a light sensor andcamera module 216, and a light module 220. The modules can be arrangedin any sequence or arrangement, in an exemplary, the microphone andspeaker module 206 can be attached or attachable to the mountingfixture, with the sensor and camera module 216 attached or attachable tothe microphone and speaker module 206. The light module 220 can beattached or attachable to the sensor and camera module 216. It can beappreciated that additional modules can be utilized with the presenttechnology. It can further be appreciated that the MDSP system 201 caninclude a general configuration allowing each module 206, 216, 220 toprovide an angular coverage suitable for the environment where the MDSPsystem is located. In some cases, the coverage can include 360 degreecoverage.

The microphone and speaker module 206 can include a microphone andspeaker array 210. The array 210 can include multiple microphones in anannular or peripheral arrangement, and multiple speakers in an annularor peripheral arrangement. The annular or peripheral arrangement of themicrophones and speakers can be provided to meet the angular coveragerequirement for that environment. A power/data port 212 can be locatedat an end portion of the module 206, which can be operable connected toa corresponding power/data port of an adjacent module.

The microphone and speaker module 206 can feature, but not limited to,an oval, curved, tapered or conical configuration with the microphoneand speaker array 210 being radially arranged. This configuration canprovide 360 degree coverage of the environment.

The sensor and camera module 216 can include a light emitter andprojector array 202, and a light sensor and camera array 204. The lightemitter and projector array 202 can include multiple light emitters inan annular or peripheral arrangement, and multiple projectors in anannular or peripheral arrangement. The light sensor and camera array 204can include multiple light sensors in an annular or peripheralarrangement, and multiple cameras in an annular or peripheralarrangement. The annular or peripheral arrangement of the lightemitters, projectors, light sensors and cameras can be provided to meetthe angular coverage requirement for that environment. It can beappreciated that a power/data port can be located at one or both endportions of the module 216, which can be operable connected to acorresponding power/data port of an adjacent modules.

The sensor and camera module 216 can feature, but not limited to, anoval, curved, tapered or conical configuration with the light emitterand projector array 202, and the light sensor and camera array 204 beingradially arranged. This configuration can provide 360 degree coverage ofthe environment.

The light module 220 can be a LED or light bulb cover, with one or morelight sources provided therewith. The light sources can be configured toprovide an angular coverage suitable for the environment where the MDSPsystem is located. It can be appreciated that a power/data port can belocated at at least one end portion of the module 220, which can beoperable connected to a corresponding power/data port of an adjacentmodules.

The sensor light module 220 can feature, but not limited to, an oval,curved, tapered or conical configuration with the light source locatedand/or the module 220 configured to provide 360 degree coverage of theenvironment.

It can be appreciated that the MDSP system 201 can have, but not limitedto a spherical, oval or any geometric configuration. The microphone andspeaker module 206 and the light module 220 can include a first endfeaturing a diameter less than an internal diameter of correspondingends of the sensor and camera module 216. The first ends of microphoneand speaker module 206 and the light module 220 are configured to be atleast partially receivable in the corresponding end of the sensor andcamera module 216, respectively. This configuration provides a modularfunction to the MDSP system 201, allowing for different modules to beoperable interconnected.

In some embodiments, the MDSP system 201 can include a track mountadapter 225, and track 224. The track mount adapter 225 can be slidablyand electrically operable with the track 224, with power and/orelectrical communication signals being communicated to and/or from theMDSP system 201 via the track 224 and track mount adapter 225. The trackmount adapter 225 can feature a hinge or pivoting mechanism, allowingthe MDSP system 201 to be repositioned or re-orientated. The modules206, 216, 220 can include a power/data port 212 located at at least oneend portion thereof, which can be operable connected to a correspondingpower/data port of an adjacent modules.

FIG. 53 illustrates how an AGUI device may implement a number ofdifferent features that are combined in a single unit 201. In thisembodiment, the AGUI hub 201 includes sensors to detect its environment,a projector, hub functionality, and networking ability. The AGUI hub maycommunicate with a AGUI Internet HUB/Server/Platform 1302, AGUI enabledhome systems & appliances 1304, an AGUI device group network 1306, AGUIenabled desktop, enterprise and mobile computing devices andentertainment systems 1308, AGUI enabled wearable, augmented reality,virtual reality and mixed reality computing, display and interfacingsystems 1310, AGUI enabled vehicles, robotics and transportation systems1312, and AGUI enabled buildings, walls, ceilings, windows, streets andsidewalks, stores and businesses, billboards, kiosks, bus stops, trainstations, airports and other indoor and outdoor displays and displaysystems 1314.

FIG. 54 illustrates how an AGUI device may implement the functionalityof a networked hub 1302 without sensor or display features. In thisembodiment, the AGUI hub 201 includes hub functionality, and networkingability. The AGUI internet hub/server/platform 1302 may communicate withcell tower/wireless communications infrastructure 1303, AGUI enabledhome systems & appliances 1304, an AGUI device group network 1306, AGUIenabled desktop, enterprise and mobile computing devices andentertainment systems 1308, AGUI enabled wearable, augmented reality,virtual reality and mixed reality computing, display and interfacingsystems 1310, AGUI enabled vehicles, robotics and transportation systems1312, and AGUI enabled buildings, walls, ceilings, windows, streets andsidewalks, stores and businesses, billboards, kiosks, bus stops, trainstations, airports and other indoor and outdoor displays and displaysystems 1314.

Other examples of an AGUI system incorporating these teachings exist,building on the concepts and description contained herein.

Reference will now be made in more detail to AGUI architecture,apparatus and operations of the AGUI system according to someembodiments. It will be understood that these AGUI architecture,apparatus and operations can be implemented in the AGUI devices, systemsand methods of the embodiments described herein including those shown inany of the accompanying drawings.

1.2. Architecture Overview

The architecture of the AGUI system is shown in FIG. 50. A localapplication 1500 is executed on an AGUI device and produces structureddisplay content. A remote application 1501 may also be executed to alsoproduce structured display content as well as receiving external events1505. The content from the local application and the remote applicationare combined to produce the structured content 1504 used by the system.A number of inputs are used by the system to configure itself. Onesource of inputs are user setup definitions 1502 which are preferencesor previous choices defined for each user or group of users of thesystem. Another is run time definitions 1503 which are decided orcalculated as the application is run. The system may start a session byinterrogating the environment and using rules to determine a choice ofdisplay surfaces. The system will remember these choices for the nexttime it is used. A user may explicitly override some or all of thesedisplay surface choices, after which the system will use the userpreferences over the rule based choices it made.

1.3. Architecture Drawing

The following statements further clarify the individual elements of FIG.50.

The hardware of an AGUI normally includes a CPU running an OS and a setof application programs. More information is in sections 1.1, 1.2, 5.1.

A program running on a remote system (another AGUI or any other device)can also provide AGUI input. More information is in sections 6.1, 6.2.

The AGUI starts with a set of programmed defaults. These can beover-ridden by the user at setup time. More information is in section5.2.

An AGUI learns its behavior by maintaining information about ongoingusage. This may result in autonomous decision making. The user canalways over-ride autonomous selections. More information is in section5.2.

Content has fixed and variable components. Variable components mayinclude options for displaying the content in different ways, dependingon surfaces chosen. Variable components may also include parameters thatare provided at run time by values from input sensors. More informationis in sections 3.2, 3.3.

Data completing display content may originate externally to the device,and be transmitted to the device over network interfaces. Moreinformation is in section 3.1.

Actual content to be displayed is completed by integrating data fromlocal sensors or from external devices. More information is in sections3.1, 3.2.

The AGUI system reviews all accessible surfaces and chooses one or morefor current display. Setup choices and run time overrides inform thisprocess. More information is in sections 4.1, 4.2, 4.3, 4.4.

Some surfaces are tied to input mapping systems. The mapping systeminforms the AGUI that the surface is available for display, and providesdetailed mapping information that will affect the display process. Moreinformation is in section 3.4.

Inputs provide numeric measurements from local sensors and mapping datafor surface selection and organization. More information is in sections2.1, 2.2.

The AGUI intelligently monitors input sensors and reacts to these inorder to maintain display integrity, deduce actions to be taken, andmodify display selection. More information is in section 3.4.

Specific content may have alternate display modes available, dependingon display capabilities. More information is in section 3.2.

112. Completed display content may be transmitted to other AGUI ornon-conformant devices. More information is in section 5.3.

113. The final step in outputting content is to render it specificallyfor the display. More information is in section 2.1.

2. INPUT AND OUTPUT 2.1. Output Devices

The AGUI system makes use of output surfaces. When used generally, theAGUI surface may refer to any active electronic display or any surfaceon which a display may be projected. AGUI surfaces may be fixed in sizeor may change dynamically. AGUI surfaces may be rigid as in traditionalflat panel displays or flexible, such as a display built on a flexiblemembrane or a flexible fabric with embedded display elements. ActiveAGUI displays may report information to the AGUI system to inform thesystem of the display's size, shape, resolution, display technology,location, communication protocol, and other parameters that the AGUIsystem can use to better use the display as an output device. A surfacemay be passive and does not report information to the AGUI system. TheAGUI system uses sensors to locate and determine the location, size,shape, texture and other characteristics of the surface to decidewhether to select the surface as an output and the best way to displayimages on the surface. An AGUI surface may also be a body part such as ahand, wrist, or forearm.

A number of common display types may be used by the AGUI system. Onecommon type is to use the open palm and phalanx areas of the fingers asdisplay regions. Alternatively, the back of the hand and back of thefinger phalanxes may be used as display regions. The AGUI system willrecognize the regions of the hand and assign functions to the regions.In some embodiment, the palm may be used as an explanation displayregion while the phalanx regions of the fingers can be used for slidersand other controls.

The AGUI may also use a horizontal surface as a display. A number ofdisplay areas can be defined for explanation regions, numeric keyboards,sliders, and other custom controls. Common examples of horizontalsurfaces are desktops and tabletops. If a vertical surface is use as adisplay, one region of the surface may be used to display a video streamwhile other regions may be used for control icons. Common examples ofvertical surfaces include walls, and household appliances. The displayregions may be adjacent to each other or may be separated. An arbitrarynumber surfaces, displays, and display regions may be supported, limitedby the usability of the user interface implemented.

Other common display types are low resolution electronic displays suchas found in televisions and computer monitor displays, either the entiredisplay or a window in the display. The AGUI is able to detect or querythe resolution of these types of displays and may choose to display lowresolution graphics, simple text frames and simple navigation icons onlow resolution displays while displaying high resolution graphics,detailed text frames, and detailed navigation icons on high resolutiondisplays.

2.2. Input Devices

Sensors may be classified into a number of types. A sensor may becoupled to or embedded in an associated passive surface, electronicoutput surface or display, providing operating parameters of the surfaceor display. Sensors can take the form of a module to detect gesturesused to input user commands. A sensor may provide data values for anambient condition such as temperature, color, light, location, as wellas biometric information on users or others in proximity to the AGUIsystem.

A variety of sensors are used to determine the bounds and content of theenvironment in which the AGUI system is operating. Inputs have standardmodels and a plug-in architecture. Sensor inputs can also beincorporated in the output display such as a pointer, gesture, orcontrol. Examples of common sensors include light and optical systems,cameras, motion, orientation and altitude sensors such asaccelerometers, gyroscopes, altimeters and compass as well astemperature, chemical, electrical, electro-magnetic, biometric, radiofrequency, wireless networking, GPS and others. Sensors allow the AGUIsystem to perform dynamic surface detection. Dynamic surface selectionis the process where the system queries the environment to detectdevices and surfaces that may be used to display information andcontrols. As one or more users of the AGUI system moves, or one or moreAGUI devices, objects, or vehicles enter, leave, or move, or objectsenter, leave, or move within one or more defined spaces, environmentsand/or locations, the AGUI system continuously detects surfaces that maybe used and updates the selected surfaces based on the availablesurfaces, the user setup definitions, run time definitions, and thestructured content to be displayed. The AGUI system adapts to theenvironment, the user, display content, and user and system constraintsand preferences.

Sensors are also used to track changes and movements in the system as itoperates. This ongoing, real time change detection takes into accountthe sensor input as well as user setup definitions to determine thecorrect action to take when the system detects movement.

The AGUI application accesses sensor input data through device drivers.Device drivers may apply to an individual device or to a class ofsimilar devices. The device drivers transform the raw sensor signalsinto an abstract language that the AGUI application can access andunderstand. The AGUI system can accept input from physical and virtualsensors. Some non-limiting examples of physical sensors are atraditional keyboard, button, mouse, touchscreen, touchpad, voice andaudio controls or a light or optical imaging system. A virtual sensor isan input derived from the manipulation of a virtual control displayed orprojected on a surface. Sensor information can include the actual, realsensor input and include keyboard and physical button action, mousemovements and clicks, and touch screen point or region selection, motionand gesture input or voice and audio controls. Virtual sensors caninclude the bounds of a selected region, control levels of a slider ordial, and identified gestures deduced from a single or multiple fingeror hand movement.

3. CONTENT AND ADAPTION 3.1. Adaption to Changing Input and OutputEnvironment

The mapping and identification of people, objects, devices andenvironments, and the subsequent mapping of interactive graphic andtextual content onto one or more surfaces may be accomplished bygenerating visual display sub-areas, typically squares or rectangles,breaking all visual content and surfaces in a camera's field of viewinto a virtual 2D or 3D grid of visual display sub-areas that may beeither physically projected using a light imaging and projection systemor be virtually computer generated. The size of each visual displaysub-area may be expanded or contracted based on the type of graphicand/or textual content and level of detail, size and definition ofcontent being projected or displayed on each surface. The size and shapeof each surface identified through light and optical depth mapping,imaging, device or sensor networking and spatial positioning and theshape, edges and boundaries of each mapped and identified displaysurface may be further calculated in relation to their position within a2-dimensional or 3-dimensional grid of squares. Each mapped surface maythen be organized into zones of one or more visual display sub-areas andinteractive graphic content may then be assigned to each visual displaysub-area or zone of visual display sub-areas based on the size, shapeand type of object and/or display surface and the type of content beingdisplayed.

If a surface is detected, the AGUI system will use captured images tomap and characterize the available surface. Mapping the surface consistsof determining the 3-dimensional spatial location of each point of theprojected pattern that can be captured in the image. The specificdetails of the embodiment of the image capture system will determine theexact methods that are used to create a 3-dimensional map of theavailable surface. In addition to mapping the surface the AGUI systemwill use the map to determine characteristics of the surface.

The AGUI system will detect the shape, viewing angle and orientation,and determine if the available surface is continuous over the field ofthe view of the image capture system. It will also detect areas wherethere is no surface and areas where there is available surface and usethis information to detect specific surfaces. An example of a specificsurface is a human hand with available surfaces on the palm and fingers.The AGUI will decide that the available surface is a hand and will usethis information when detecting surfaces, selecting surfaces, adaptingcontent to surfaces, and detecting changes in surfaces. If there are no‘holes’ in the available surface and it is essentially flat and there islittle if any distortion in a projected and imaged pattern plus thedirection of projection and image capture is horizontal, the AGUI canjudge that the available surface is a wall or vertical surface. If theavailable surface is mapped as essentially flat but the pattern has acharacteristic keystone distortion and the direction of the projectionis somewhat below horizontal, the AGUI system will judge that theavailable surface is likely a table top or some similar mostlyhorizontal surface. An AGUI system will be aware of a number of commonsurfaces that can be used and will remember and learn surfaces that werepreviously used in a particular location or environment.

If the AGUI system finds an available surface and in addition recognizesthat this surface is associated with a device with which it cancommunicate and that has the ability to create a display or includes adisplay, yet another type of available surface will be found.

Interactive graphic content assigned to a visual display sub-area orzone of visual display sub-areas may freely move within the gridenabling content to continuously adapt to stationary or moving objectsand display surfaces and/or enable one or more moving projection ordisplay systems to adapt content in relation to one or more stationaryor moving objects or other display surfaces. Content may alsodynamically adjust in size, spatial position and orientation within eachvisual display sub-area or zone in relation to the spatial position andviewing angle of one or more viewing parties.

FIG. 51 illustrated the process of detecting and utilizing displaysurfaces according to one embodiment of the present technology. Aprojecting AGUI device with an integrated light pattern projector willproject a light pattern within an area of the environment where asurface might be found. This can be a randomly chosen area, an areaindicated by a user, a pre-defined area, or an area where a surface waspreviously detected. A detecting AGUI device, which may be the samedevice as the projecting AGUI device or a separate device, will detectthe light pattern and use this information to detect any surfaces in thearea under examination. The AGUI system will determine if the detectedsurfaces can be projected on. For example, a white or gray wall iseasily projected on. A dark, shiny, uneven, or glass surface is likelynot. Electronic surfaces such as a computer monitor are often reflectiveand would be detected as a non-projectable surface. If the systemdetermines that a surface is non-projectable it next determines if thesurface has the ability to create a display. This will typically involvequerying the display using electronic or communication protocols as arewell known in the art. If the AGUI system is able to communicate withthe surface it will query it for information such as resolution, colordepth, supported interfaces and protocols. If sufficient information isavailable, the AGUI determines that this surface is an electronicdisplay, external or internal to the AGUI system, and may then use itfor display.

For surfaces that are determined by the AGUI system to be projectablethe system will then use images of the projected light pattern to mapthe surface and characterize its surface. The AGUI system will adjustand optimize its projection settings to produce an optimal display on asurface. This includes, choosing a resolution, adjusting the color andbrightness, adjusting the shape of the image to account for the viewingangle of any viewers, adapting for the roughness, and surface texture ofthe surface, and adjusting for irregular shaped surfaces or surfaceswith non-projectable areas or holes in them. The combination ofprojection settings may be grouped and saved as projection scenariosthat can be recalled and reused on the same surface or similar surfaceslater. Once detected and characterized the AGUI system creates a displaybased on a current display scenario, surface, mapping of elements to thedisplay, and the characteristics of the current display or displays.

While the AGUI system is operating, it continues to monitor the imagequality on the display surface. By using its imaging system, the AGUIdevice or devices will compare the current image to previous images andcontinuously make adjustments in the projection and display settings toobtain an optimal image for the use. At the same time the AGUI systemalso monitors for control movements; the selecting or movement of acontrol, a gesture, or an input. When a control is detected the AGUIwill perform the appropriate action.

FIG. 52 provides further details of how the AGUI may query and interactwith electronic displays. The process assumes that communication hasbeen established between the AGUI system and the external displaydevice. The AGUI will query the external display to determine if it canself-map its surface. A self-mapped surface is one that can determinethe position and viewing angle of display elements with respect to theuser or viewer. If the display cannot self-map its surface the AGUI willdetermine if it is able to image and map the display surface. If this ispossible, it will map the surface and chose a display scenario to bestuse the surface based on the position of the viewer. Then proceed tocontrol and update AGUI process. If the display surface has the abilityto self-map, the AGUI will attempt to establish the orientation of theviewer to the display surface. If the orientation with respect to theview is successfully determined then the AGUI system will choose adisplay scenario to best use the surface based on the position of theviewer. It will then proceed to control and update the AGUI process tobe aware of the scenario. Similarly, if the AGUI system cannot image andmap the surface but can establish the orientation of the viewer to thedisplay surface using other mean, the AGUI system will again choose adisplay scenario to best use the surface based on the position of theuser. If the orientation cannot be determined, a scenario that assumesthe use positions themselves in front of the surface will be chosen.

The remote application may trigger external events that are merged withthe structured content. Input may also be received from networkeddevices and networked data sources. These inputs, together withinformation associated with the selected surfaces and real time changeinformation are used to generate content based on the selected surfaces.The content is modified based on display type and an output stream ofdisplay data is generated to be sent to the selected surfaces.

The AGUI application produces a stream of output content. The AGUIsystem may also accept external events from a remote application whichcan include user input. Display content may be supplemented with sensorinput data such as the user's temperature or heart rate. This caninclude input from the environment in the form of static sensors orsensors embedded in objects that may be manipulated by the user.

3.2. Content Adaption to Surfaces and Surface Response

In the AGUI system, an application determines display content. Contentconsists of frames. Each frame connects to at least one other frame viaevents that the application can recognize. Examples of events are a useraction such as a gesture, a mouse click, a keyboard entry. Otherexamples include system occurrences, such as a change in mapping datainput, a change in system setup, or an external request from anotherAGUI device.

When an event occurs, the system uses a lookup mechanism to determinewhat action should occur as in consequence. It may result in a change tothe current frame, such as modifying a data field with a new inputvalue. It may also cause a change in display dynamics. The applicationincludes rules that control changes in display activity. Rules exist ina hierarchy, lowest to highest priority. The lowest priority rules aresystem defaults. Next are user adjusted thresholds that specify howlarge a change in an input or combination of inputs will cause an actionto occur. Next higher priority are learned methods, that is, at runtime, when the user overrides a certain programmed behavior a certainnumber of times, and subsequently the system remembers and the learnedmethod becomes a rule. Highest priority rules are run time overrideswhen the system provides a way for the user to interact and change anautonomous selection.

The system monitors certain input values to adjust the displayed image.When the numeric values of a set of input values enters a certain range,a rule is triggered. The computation to determine whether a changeshould occur may be simple or complex. In a simple situation, a changeof ambient light level might result in increased or reduced brightnessof the display. In a situation of medium complexity, changes to themapping data might indicate a shift or a resizing of the displaysurface. Again, the application consults rules to determine whether thisshould cause scaling or positioning of the display content to change tocompensate for a change in order to maintain a steady display. Suchgeometric change possibilities also accommodate rotational changes inthe user and surface relationship. In some cases, the rotation is in thesurface plane, in which case the geometric transformation that must beapplied to compensate for this is a linear transformation. If thesurface tilts, a non-linear transformation that accounts for differentsurface elements being closer to or further from the user may result. Insituations of further complexity, displaying content on non-planarsurface elements of the surface results in algorithms that are aware ofthe non-planar properties. These situations include, adjusting for localcurvature of a surface, raised or lowered sections, and situations thatare hard to characterize with simple descriptors, and are better handledby a spatial compensation table.

A frame may have different styles for displaying its content on somepossible surface types. It is not necessary for a frame to have adisplay style for every possible surface type. Rules allow the system tomake a choice between available display styles when a clear winningdisplay style does not exist among the encoded possibilities. The systemalso has defaults that guide a primary choice. The user can over-ridethe default behavior at setup or at run time. FIG. 7 shows the mappingsin one normative system. Others are also possible. When the display typeis a hand with palm and finger zones, the content may be pre-organizedinto an explanation region to display on the palm, selection regions todisplay on the fingers or parts of the fingers, and slider controls todisplay on fingers. In the case where the display type is a horizontalsurface, the content may be pre-organized into an explanation region onone part of the surface, a numeric keyboard on a second part of surface,and sliders or custom controls on other parts of the surface. Inembodiments where the display type is a vertical the content may bepre-organized into video stream on one part of surface and selectionicons on another part of the surface. For a display type comprising adisplay built into clothing, the content may be a single combinedgraphic or text frame. In the case where the display type is alow-resolution window displayed in a TV or computer display the contentmay be low resolution graphics with simple text frame and navigationicons. Alternatively, where the display type is a high resolution windowdisplayed in a TV or computer display the content may be rendered ashigh resolution graphics with detailed text frame and navigation icons.

The AGUI application is aware of the set of output devices and displayscurrently in use, and optimizes the content for those devices,generating a set of virtual output streams. The output streams aresubmitted to the specific device drivers. The device drivers perform themapping from virtual streams to device-specific actual streaming datasent to the devices. The executing application owns a stream of virtualoutput content. Rendering this output, advancing the flow of theapplication, and loading a different or modified application istriggered by several events such as when new output surfaces aredetected and existing output surfaces become unavailable, the user makesa selection, a sensor provides data that changes program flow, or asensor provides data that completes an output display frame.

The AGUI system is provides clear output content even when the systemcomponents, environment, or parameters change. The AGUI recognizes therelationship between the user and the output surfaces, and adjusts theoutput accordingly. This adjustment makes use of all the relevantavailable input data that concerns the user, the surface, and the waythe user views the surface. As an example, the AGUI makes constantcorrections for perturbations of the geometry of the display surfaces orenvironment. If the user's hand moves, the content moves with the hand.If ambient light level changes, the AGUI amplifies or attenuates theoutput as appropriate. The AGUI compensates for detected surfaceirregularities, including curvature, color and texture. Shapeirregularities include clothing shift and specific finger and palm areashape, and include allowance for computed dead zones where projectedimages cannot reach.

When the device displays onto a human hand, the camera system mayidentify fingers, determine the geometry of the hand and fingers, andmap parts of the content onto the palm and each finger, providing a menuor control system. Detected perturbations of the fingers allow the AGUIto make corresponding geometric adjustment of the content so thatcoverage is optimized and clarity is maintained. Additionally, a gesturefrom a finger may be recognized. Depending on defined system setupchoices, the gesture may be applied to the processing in different ways.Processing examples include a change to a virtual device such as avolume control slider for an audio subsystem, or a menu selection thatadvances static or streaming video output content.

When the device displays onto a large continuous surface such as a wallor table top, the device input provides surface response data, surfacesize and geometry, and location and orientation. Surface response datamay include color and texture, allowing the device to optimize outputfor clarity. Surface size and geometry inputs allow the device to modifycontent to fit the display area and adjust for non-planar surfacedistortion. Location and orientation input may trigger content choicessuch as the display of a virtual keyboard on a horizontal surface orstreaming video content on a vertical surface.

When the AGUI system creates a display by projecting a colored displayonto a surface the AGUI system will dynamically control the colors sothat the viewer will see the desired colored display. This is done bycontinuously capturing an image of the display and comparing theindividual pixel intensities to the metameric color match desired. For agiven 3 color sensor system, such as a CCD or CMOS camera, the spectralsensitivity functions of the 3 types of sensor are known. Using thesespectral sensitivity functions, 3 captured intensities for each pixelare then converted to a metameric value. If the metameric value is notthe desired one, the AGUI system will command the individual pixels ofthe projected image to increase or decrease intensity until the image iswithin tolerance of the desired metameric color. It is recognized thatthe color perceived by the viewer will be influenced by how the surfacebackscatters the projected light, the color of the surface and by thebackground illumination that falls on the surface and does not come fromthe projector. However, by first measuring the net effect of theprojected light, the amount of backscatter, the color of the surface andbackground illumination, then increasing or decreasing the projectedlight and monitoring in real time the effect of these commanded changes,the AGUI can dynamically adjust the perceived color until it is withinthe desired range for each area of the projected display.

In some embodiments, the AGUI may project onto a flexible surface suchas a fabric surface. When the AGUI establishes communication with aflexible self-luminous display system, such as a fabric or a garmentwith implanted OLEDs, it will determine the level of surface mappingdetail that the self-luminous display system can provide to it. Thereare several types of fabric that provide different sets of informationto the AGUI system.

The most complete set of data that a flexible self-luminous displaysystem can provide consists of a set of three spatial locations for eachluminous display element in the coordinate system of the display systemand two orientation values for that coordinate system. The coordinatesystem is typically given orientation by assigning one coordinate tolocal vertical and assigning a second coordinate perpendicular to localvertical in a universally common direction, such as north. This type ofa flexible self-luminous display must be able to create a spatial netgiving the locations of all luminous elements with respect to oneanother and have the ability to sense local vertical and a universallycommon direction. If the AGUI system can sense its own orientation whenits image sensor is directed at the flexible self-luminous displaysystem, it will use the information received from the flexibleself-luminous display system to create an internal map of the availabledisplay surface and then use this map to command the desired display tobe formed by the flexible self-luminous display system.

Another type of flexible self-luminous display system is only able toprovide a set of three spatial locations for each luminous displayelement in the coordinate system of the display system. If the AGUIsystem can capture an image of the flexible self-luminous displaysystem, it will perform a 3-dimensional coordinate transformation on thereceived data until the transformed display matches its capture imagewith sufficient accuracy. The AGUI system then has a map of theavailable display surface and will use this map to command the desireddisplay to be formed by the flexible self-luminous display system.

At the lowest level of complexity, the flexible self-luminous displaysystem can provide no spatial information on position of displayelements as the surface flexes. It can only provide static locationinformation on the display elements so that they can be controlled tocreate a display. If the AGUI system can capture an image of theflexible self-luminous display system it will initially command thedisplay to create a special referencing display and compare thiscommanded display to the captured image of the display. The AGUI systemwill then perform a 3-dimensional coordinate transformation on thecommanded display so that it matches its capture image with sufficientaccuracy. Thus, the AGUI treats the flexible self-luminous displaysystem as it would a non-flexible 2-dimensional display. The AGUI systemthen has a map of the available display surface and will use this map tocommand the desired display to be formed by the flexible self-luminousdisplay system. Then the AGUI continuously captures images of theflexible self-luminous display and uses them to update the initial mapof the display, in this way dynamically accounting for surface flexureand changes in surface orientation with respect to the AGUI system.

If the AGUI system does not have the ability to image the flexibleself-luminous display, it will assume that the viewer is oriented mostfavorably to view the display and will create a map of the display basedon this assumption. Then the AGUI system will command the desireddisplay to be formed by the flexible self-luminous display system.

A user manipulates the AGUI system by manipulating an active zone on adisplay. A potentially active zone on a finger may be selected foractivation by a finger of another hand, or by a perturbation of thezone, such as a flicking motion. A finger covering a selection site on apassive display may provide the same activation.

Intelligence surrounding the projector's display capability adapts thedevice's output content to take advantage of the organization of thesurface into zones. When the content comprises a user interface thatincludes selections, those selections may be organized onto the fingers,and explanatory static or dynamic video content is assigned to the palmarea. Processed mapping data recognizes that finger zones includesub-zones separated by joints, and the device can choose to furtherdivide selection content onto specific parts of fingers. When thecontent includes controls that map to functions with a continuouslyvariable quantity, virtual controls conveying this function are alsomapped to finger zones. When the content includes alpha-numeric dataentry, such as a keypad for entering numbers, or a graphic interactiveicon array, photograph, video or other interactive media, content isdisplayed on the palm and fingers, wrist or other body surface orsurrounding objects and display surfaces.

If the device remains attached to the user, but the user has reorientedor moved the device, corresponding large changes are noted in the inputmapping data. One of the following different display scenarios mightthen be identified. If the device is flipped, the mapping systemidentifies the back of the user's hand, consisting of regions as in thecase of display onto the palm and fingers. The software again adjuststhe output content to accommodate and make best use of this new surfaceorganization. The device can be slid along the user's arm. The mappingsystem then discovers an additional larger continuous display areaconsisting of the user's clothing or skin area or a combination. Sensorinputs provide information about the response of the surface and itstexture and color composition. These inputs permit furtherreorganization of content and adjustment to the parameters of display,so that content is geometrically mapped to match the surfacerequirements and adjusted for response variation over the surface. Thedisplayed content remains clear and readable. Finger gestures from theuser's other hand interacting with the display content may provide userinterface input.

If the device is detached from the user, its mapping system searchesfurther afield for passive display surfaces. The device is constantlysearching for display surfaces that can be utilized. Sometimes these areautomatically selected and brought into the device's set of currentlyavailable surfaces. At other times, the device may request that the userbe involved in the selection of output surfaces from those available tothe device. Examples of passive surfaces include a table top, a wall, oran appliance.

The device also has access to input data from sensors such as agyroscope to give it an orientation reference, and can thus distinguishbetween a horizontal table top and a vertical wall. In an analogousmanner to the device's display reorganization for a hand, the devicemakes decisions about how to best organize the content onto suchsurfaces. The mapping system and other sensors further guide the devicein this activity. Inputs that are taken into account for this purposeinclude surface curvature, so that this can be compensated for clarity,surface orientation relative to the projector, so that the desired shapeof the content is actually projected onto the surface, surface detailedresponse characteristic, so that surface distortion can result in shiftsin the content to minimize the distortion's impact on clarity, andsurface color composition, so that color can be remapped such thatcolors do not disappear.

The examples up to this point have all required the built-in projectorto illuminate the display surface. Resolution is mainly a function ofthe projector. The type of surface also influences effective resolution.A smooth surface allows the projector to work simply. A rough surfacemay require compensation. The roughness can be modeled with the aid ofmapping system input data.

The output of a single built in projection unit is limited to what is inthe projector's display field. However, the device's output capabilityis not limited to what can be projected directly. The device can alsoconnect via wired or wireless interfaces to other kinds of displays,including a front panel OLED display on the device itself, a flexibleOLED display built into fabric attached to or in proximity to the user,a computer running a display application, a TV that assigns an inputchannel to the wrist device output, and another AGUI device.

The device makes value judgements, in keeping with the user's predefinedexpectations, about content modification. When the surface is thedevice's front panel OLED array, the content may be displayed withselection icons. When the surface is a fabric-based OLED array, thecontent might be modified to enhance the user's personal appearance.When the device connects directly or wirelessly to computers ortelevisions that recognize the device, the display unit provides itsscreen or a window of its screen to be used for the device's output. Thedevice's general approach to making use of the surface does not change.The device will choose content and adjust it accordingly. In ananalogous manner to the way the device receives mapping data, the sizeand shape of the display window and the display resolution are madeavailable via the device's input sensors. These factor into the device'sdecisions about how to map content to a surface. A large high resolutiondisplay might inspire the device to generate detailed content, includingtext and graphic explanations. A low resolution display might result inless detail being shown. If the window has an unusual shape, this factis available to the device, and might also cause the device to truncateor reorganize content to match that shape. AGUI systems comprising awearable projector may have the ability to adapt content in real timebased on the position of the user. The AGUI device comprises a wearableprojector to display images on target viewing and interfacing surfaces.The system actively adapts projected content to one or more surfaces byassigning different projected content to one or more projectors based onoptimum viewing or display angle of available projectors and/orwirelessly networks with other projection or active display systems. Itoperates as a motion and orientation sensor to identify the wearer'sposition and orientation relative to surrounding objects and displaysurfaces.

In other embodiments, an AGUI system operates to implement aheads-up-display. This may include the ability to identify wearabledisplay types such as a binocular head-up display and autonomously adaptthe user interface to a binocular, immersive display environment.

3.3. Displaying onto Vehicles and Other Moving Surfaces

Other embodiments of the present technology include the display ofcontent on a moving surface. In a situation where the spatialrelationship between the user and the surface only experiences smalllocalized changes, data from the mapping system provides the necessaryinformation so that the device can make small changes to the displayoutput to accommodate those changes, and maintain clarity and lock inposition. In a more complex situation, the user may be moving or thesurface may be moving or both the user and the surface may be moving. Ifmapping data exists that can identify changes in the relative positionand orientation of the user and the surface or display, algorithms canbe run to adjust the content geometrically to maintain display clarity.Typically, the display surface can be imagined as an array of smalltriangular regions. If content were sent to the display uncorrected, theuser would experience the content as geometrically distorted. If themapping data can be used to model the distortion as a transformation,the inverse of this transformation can be applied to the content beforeit is displayed, and the user will see the content undistorted. Thisinverse transformation is recomputed constantly, and correspondingcontent adjustment is modified in real time so that the user experiencesa display that appears not to be shifting.

In some embodiments the position of one or more viewers comprises theirviewing position outside of a vehicle in relation to one or more displaysurfaces on the outside of a vehicle.

In other embodiments the device dynamically optimizes the display acrossone or more moving or stationary vehicles or other display surfaces suchas billboards, buildings, store fronts, bus stops, ground, road, streetand other display surfaces based on the spatial and relational position,location and orientation of one or more of the vehicles display surfacesto one or more display surfaces on one or more other moving orstationary vehicle display surfaces or other display surfaces.

In other embodiments the device dynamically optimizes the display acrossone or more moving or stationary vehicles or other display surfacesbased on the spatial and relational position, location and orientationof one or more viewers based on each viewers viewing angle to one ormore vehicles, vehicle display surfaces and/or one or more other movingor stationary display surfaces and/or based on the spatial position andorientation of one or more viewers to each other.

Display optimization may be accomplished using one or more sensors forimaging, mapping and/or identifying the distance, spatial position,location and/or orientation of one or more vehicles, vehicle displaysurfaces and/or other display surfaces to one another and/or foridentifying the distance, spatial position, location, orientation and/oroptimum viewing angle of one or more viewers to one or more vehicles,vehicle display surfaces and/or other display surfaces. This may also beaccomplished using motion and orientation sensors on the vehicles anddisplay surfaces combined with location and wirelessly networked datafrom one or more wirelessly connected vehicles and/or display surfaces.One or more viewers may also be identified and mapped based on theirrelational position, location, orientation, viewing angle and/or pointof focus of focus in relation to one or more vehicles or other displaysurfaces by wirelessly connecting with one or more viewers connecteddevices or sensors such as mobile handheld, carried, wearable, embedded,attached, implanted and remotely monitoring devices, clothing and othersensors assigned to one or more viewers.

3.4. Creating Virtual Controls on Surfaces

As well as the embodiments discussed above, embodiments of the presenttechnology may be used in a variety of applications, of which only a feware listed here.

The AGUI system may comprise a ceiling, wall, table or other surfacemounted single or multi-directional mapping, imaging, projection and/oractive display. It may also include an interfacing, networking andmultimedia hub capable of depth mapping and imaging a room or otherenvironment. As the environment is mapped, the system identifies people,objects and display surfaces, and dynamically and continuously adaptsinteractive graphic content to one or more passive and active displaysurfaces. This is based on the type of object or device, whether thedisplay is an active or passive display surface. The system alsoconsiders whether there is autonomous or user controlled contentselection and assignment of interactive graphic content to one or moreobjects and surfaces. The adaptive display system may assign a uniqueinterface or graphic content to specific objects and surfaces based onobject type, shape, function and application. The AGUI display systemmay also adapt graphic content based on the relational position of oneor more viewing parties in a mapped environment. This can be based onthe preferences of one or more identified viewing parties so that eachviewer may view content specific to user preferences or specific to thedevice or object on which the graphic content or interface is assignedor displayed. The system may also dynamically move graphic contentacross multiple passive or active display surfaces or shift content fromone display surface to another based on user selection or based on oneor more users' activity, movement, position, orientation, direction andpoint of focus (where the person is looking) within a mappedenvironment. The adaptive display system may also select the optimumdisplay surface and position, size, content and optimum viewing anglefor one or more people based on the position of each person within amapped environment on one or more display surfaces.

Virtual controls may be mapped to a variety of activities including aninterface to front panel controls of a vehicle, adjusting the audiolevel for headphones, adjusting the temperature level of a thermostat,entering remote commands for a TV, and telephone number entry. In orderto identify the selection of all the activity described above, themapping system is constantly re-evaluating input data. Small datachanges result in this activity being carried out. When signal changecrosses a threshold, the system may identify a change of displaysurface. The device is constantly reviewing the mapping system signalsand comparing signal patterns with patterns that would identifydifferent surfaces.

3.5. Interpretation of Inputs (Small, Control, Surface Change)

Once the AGUI system has found, mapped and identified an availablesurface and used this information to select an appropriate AGUIscenario, the AGUI system will enter an active display and respond mode.It will project onto or command to be displayed on the surface visualcontrol icons or features. At the same time it will use its imagecapture system to monitor the available surface and field of view formotion of various types. Once the available surface is mapped, the AGUIwill be able to create a projected image or generate commands to displayan image that is known to create a desired image at specific locationsin 3-dimensional space. As the AGUI captures images of this display itwill continuously compare the locations of the image features to theirexpected positions and use any difference in location to update that mapof the available surface. These changes in the locations will bereferred to as small motion detection by the AGUI system and are used tocontinuously update the mapping. At the same time the updated map isused to generate the current projected or commanded display so that itis always on the available surface in the desired locations. The AGUIsystem has the ability to judge if the available surface or the currentportion of the display process has changed to the extent that thecurrent display configuration is no longer appropriate. If this is foundto be the case, the AGUI will reconfigure the display so that it is bestfor the current available surface.

In addition to the small motion detection performed by the AGUI system,it also analyzes the captured image for control motions. Depending onthe current scenario, certain movements are recognized by the AGUIsystem as control signals. If such movements are detected, the AGUIsystem will cause the system it controls to take appropriate action. Thesystem may also change the display in response to the recognizedcommand. For instance, if the AGUI system is embodied in a wrist mounteddevice with the current scenario being a display on a human hand andwith a specified control icon projected onto the end of the middlefinger of the hand, the thumb is observed to move over this icon andremain there, this may signal the AGUI system that the user has selectedthe control action associate with the icon. The AGUI system will thencommand the appropriate action by the system it controls to be taken,and in certain instances, change the display projected onto the hand. Onthe other hand, with the same scenario and display situation, if thethumb is observed to move to the middle finger icon and then move to theend of the index finger, this may signal the AGUI system to move theicon from the middle finger to the index finger without any othercontrol action being taken. This type of interactive control isprogramed into the AGUI for the various scenarios it can expect andthese controls can, of course, be altered by a person authorized tochange the way in which the AGUI acts.

Another situation occurs if there is such a large change in theavailable surface that the selected scenario is no longer appropriate.In that case, the AGUI process will revert to the mapping and surfacecharacterization portion followed by a new scenario selection phase.

Two types of movement need to be considered when analyzing monitoredimages created by the AGUI system. When the device moves with respect tothe imaged space all features in the image move together, a movementknown as a translation. This type of movement will continuously happen,whether there are small vibrations or larger movements. A second type ofmovement is the movement of the user's hand or portion of the hand withrespect to the display being imaged by the AGUI system. This is not atranslation but is the type of movement associated with a potentialcontrol movement and is local in nature. The AGUI can distinguishbetween these two types of movement. This can be done by a correlationprocedure in which a correlation computation is done between sequentialimages with iteration to see if correlation is higher in an overallimage translation of one image with respect to the other. Having foundthe translation with the highest overall correlation, this translationis applied locally. Local areas where the correlation markedly decreasesare areas where local movement has occurred. These are local areas wherea control motion may have been made. To assess this possibility thedisplay content in the local area is assessed. If this area of thedisplay is a control feature that can be commanded with a movement, theAGUI system considers a control action command to have been received andacts accordingly. This process is enhanced by the AGUI system by findingin the captured image hand and finger locations. The AGUI system canthen use this additional information to compare the hand and fingerlocations to images of control features and give especial attention tonon-translation motion in these areas.

To better understand how the above process is implemented the followingexample is given. The first step in the process is to account for anytranslation that has occurred in which the camera system itself moveswith respect to the scene it is imaging. As the imaging system capturesimages, it compares the intensity of each pixel in the image just takenwith the intensity of the same pixel in the previous image. Thesedifference intensity arrays are continuously formed and updated witheach new frame captured. If there is no movement at all between frames,each difference value will be zero or an insignificantly small number.If there is movement this will not be the case and by summing thedifference and detecting a non-zero sum the routine will sense amovement of some type. The procedure used for this is known as acorrelation procedure. The routine then shifts one array of intensitieswith respect to the other in both dimensions, performing a correlationat each step and finding the shift corresponding to the best correlationbetween the image and its shifted copy. As this is done for eachcaptured frame, the time between capture is very small and so imageshift to get the best correlation for the overall image will typicallybe only a few pixels. However, the shifting procedure is done until eachpixel difference at some step has close to a zero value. This is done sothat information is recorded for use in finding location movement. Asthe above procedure is done, the difference array saved for each shiftstep is labeled with the horizontal and vertical shift associate withit.

Of the shifted differences taken, the one with the best correlation isthe one in which the translation effects have been removed. For thisshifted difference array, areas that have higher difference valuesremaining are those where control motion may have occurred. In each ofthese areas where the AGUI has created a display control icon in theimage the AGUI will examine all the saved difference arrays for thatarea and find the one with the lowest difference value. The differencein the pixel shift values for the translation difference array and thepixel shift values for the shift that removes overall translation givesthe movement size and direction for that area within the scene. Byrecording the movement in display control icon areas over severalframes, the AGUI system can tell if a control motion had occurred andtake appropriate action. A control motion is a motion with sufficientmovement in a relevant direction.

It can also occur, in some situations, that a finger or hand gesturewill constitute a control motion where there is no display control iconby it. To account of this possibility, the AGUI system may use, insteadof a display control icon location to check for motion, the mappedlocation portions of the hand. This will require the AGUI system toinitially perform an image recognition function to find fingers andother hand parts in its field of view. Having initially done thispattern recognition, the image motion detection process will becontinuously used to update the hand feature locations in the memory ofthe AGUI system. Then for each frame correlation with the previousframe, all the necessary location information is available to look forand use motion information.

Thus the scene will be intelligently painted onto the user's hand andfingers. In the same way that a conventional user interface now waitsfor a user command from a mouse, touch screen or keyboard, the AGUIreacts to the user's input in a variety of ways. The user may manipulatethe interface using a finger of the other hand. The user may touch avirtual control to make a selection. The user may use a finger gesture,such as a flick of a fingertip, to make a selection. The user may slidea finger of the other hand over the control area of a virtual volume orbrightness control to change the control's level. The user may tap thedisplayed virtual key of a keypad to enter that value.

Finger gestures are recognized via perturbations of data from themapping system. The data is recognized as a finger. Perturbation datatracks its location. Localized changes to zone extent or localizedcontinuously variable signal return path length imply a finger gesture.Localized changes to signal path in a certain area imply a selectionfrom a finger of the other hand.

Selection items may advance the user interface, or may be mapped vianetworking to external devices. Advancing the user interface includeschanging the content displayed and/or the way it is organized onto thepalm and fingers. For example, a finger from the other hand might dragcontent from one finger to another, or from a finger to the palm area.Or a finger might tap a selection zone to activate a change in the wholedisplayed scenario.

4. WORKING WITH SURFACES 4.1. Surface Selection

The AGUI recognizes available electronic displays through wired andwireless connections, and through standard and proprietary protocols.Via these protocols, the AGUI receives display parameters from thedisplays, and negotiates with recognized devices for availability fordisplay. A device that is available sends the AGUI its displayparameters, including physical dimensions, pixels dimensions, pixelspan, color resolution.

A standard display device may already run a standard protocol to allowit to connect to the AGUI system. In the simplest case of a directlywired display, this data is available immediately the wired connectionis made, via that wired connection. In a more complex case of a wirelessconnection, pairing activity may have to take place prior to this.

A display device that does not run such a protocol natively can be madeavailable to the AGUI by being provided with a custom protocol, whichcan be downloaded to the device or run from a dongle plugged into thedevice for this purpose. The purpose of the dongle can be to activate aprotocol that is already resident on the display device, or to providethat protocol at run time. An example of this would be to plug an AGUIenabled HDMI dongle into a television or computer monitor that does notnatively support the AGUI system.

Each surface has the capability to interact with the user in a uniquefashion. The extent of the interaction depends on the completeness ofthe available scenario data. A scenario is a set of data that describesthe environment and use of the AGUI system. The algorithms for selectiontake in the full set of available scenario data, merge this with theuser's setup selections and preferences and the user's actual usagelearned over time, and apply overall computed preferences to thecontent. If content has the capability to be displayed in different wayson a single display, the usefulness of that is taken into account. Forexample, if the user is known to be in close proximity to a table top,and if the content contains material that can make use of a table topuser interface, the AGUI is encouraged to display that user interface.In general, when decisions about content choice and content distributionis made, a choice that favors user interaction is encouraged. However,this also depends on how the user has set the device up, and how theuser has operated the device in the past.

When multiple surfaces are deployed in proximity to each other, the AGUItakes into account whether coordinate data for the surfaces suggeststhat they could together constitute a large continuous surface. Whenthis happens, the AGUI may make this a strong preference among alternatedisplay mode possibilities. In this case, the content is spread acrossthe surfaces.

On the other hand, the multiple available surfaces may be disjoint andlack the kind of connection required for continuous display. The AGUImay then choose to repeat the content on each of the multiple displays.When this happens, the AGUI still tailors each display's actual contentto the properties of the surface, including reorganizing content toaccommodate non-displayable regions.

4.2. Surface Mapping

A specific display type is an array of OLED elements embedded intoclothing. For displays of this type, each OLED announces its coordinatesso that the AGUI can build a 3D map of the overall surface that can beviewed by a person in proximity to the clothing OLED array. Thecoordinates will include the location of each display element and mayalso include an indication of the viewing angle of each element. Thisanalysis includes awareness of OLEDs that are hidden from view. The mapis constantly rebuilt as the array shifts with time. When assigningcontent, the AGUI is aware of OLEDs that are not visible, and these areexcluded in the geometric transformation of the content therebyimproving performance. Content may contain within its structure theability to display in different modes. The display mode can also beaffected by input sensors that provide context information. The AGUIalgorithms include scenario-based choices and geometric corrections.

In one example, sensor input data indicates to the AGUI that its outputsurface is a human hand, and that a sub-mapping of the hand is possible.The sub-mapping may identify zones representing individual fingers and awrist or palm area. The AGUI may then take the display data stream andsplit it so that specific parts of the display content are directedtowards specific zones. Fingertips may be identified as menu selectionpoints. The wrist or palm area may be used for static or dynamiccontent.

As another example, via sensor input data the AGUI may recognize thatits output surface is a passive flat surface, such as a table top or awall. The AGUI performs a different organization of the content, onebetter suited to the different properties of that surface. In bothexamples, the AGUI can also paint virtual controls such as sliders thatcan be operated by a finger or gestures.

The AGUI discovers and detects passive surfaces and the detection systemsensors identify physical properties related to displaying on thesurface. These include the presence of the surface, the spatialseparation, orientation, shape and complexity, surface resolution,surface response, surface type, surface properties, and whether thesurface is a 2D or 3D surface. The AGUI also detects the geometricalrelationship to the user and viewing angle of the user, properties ofthe surrounding environment, and real time perturbations of separation,orientation, shape and response of the surface. Input and output areoften tightly bound together. A passive output surface makes use ofinput sensors to establish both geometrical and response characteristicsof the surface and its surface's relationship to the projector. At theapplication level, the AGUI sees input and output peripherals asabstractions delivering or consuming virtual signals. The applicationdoes not interface directly with peripherals, but interfaces through anabstraction layer designed so that peripherals conform to a commoninterface model. This abstraction layer captures each peripheral'sproperties that are essential to the application's activity.

A method to determine if a detected surface may be used for projectionsis for an AGUI system that includes a projection and imaging sensor toonly project a recognizable grid intermittently while continuouslycapturing frames that include the surface. The image of the frame wherea pattern was projected is compared to the previous image frame where nopattern was projected. If an intensity change is found, there exists asurface in the field of the projector that can back scatter light and sorepresents a surface onto which a display can be projected. This could,for instance be a wall, an article of clothing or a table surface.However, a glass or reflective surface such as a computer display screenwould in almost all cases reflect light so that it would not reenter theimaging system of the device and so would not represent a surface ontowhich displays could be projected even though it might well be a surfacethat could create a display and could participate in the system AGUIsystem if the system could communicate with it.

If no projectable surface is found, there may well be an active displaydevice that could be used as an AGUI display in the field of view of thedevice image capture system. This type of surface can be detected byanalyzing the image frames other than the one coincident with theprojected pattern with known feature detection methods to look for textor other features characteristic of computer displays. If the likelihoodof such features in the captured image is sufficiently high, the AGUIsystem will assess if it is also in communication, or could be incommunication, with a device that is likely to be able to create adisplay. If this is the case, the AGUI process will attempt to connectto the display, to obtain configuration information, and to use thedevice as a display.

The AGUI displays onto an active or electronic display via a direct orwireless connection. An active display is characterized by configurationdata received from the display itself or from an external source oncethe display has been identified. Other characteristics of the displayare determined by the AGUI system. Characterization of the displayconsists of detecting the presence of the surface, the image or pixelresolution, surface type, physical dimensions, physical shape, and colordepth. The system also takes into consideration environmentalcharacteristics such as ambient light, areas of bright light or shadow,and the distance of the viewer from the display and their viewing angle.The system assumes that any of the display properties can change at anytime and monitors the display for changes.

Display and surface interfaces may be abstracted through the use of datastandards, giving the AGUI the capability of flexible operation. Sensorinput is abstracted. Output surface and display specifications arestandardized. The AGUI application drives any given output surface witha set of virtual video streams. The driver for the specific deviceperforms the mapping of virtual streaming data to actual data thatdrives the device hardware. In constructing the virtual video stream,the application leaves compensation for surface type, spatial andresponse to the device driver. The application's virtual videoconstruction takes into account a number of characteristics of theoutput surface such as surface shape data. The surface shape may be flat(a planar surface), a known kind of curvature, an irregular shape (via amapping table). Other characteristics are surface type—electronic orpassive; surface size; surface topography—flat, circular, elliptical,parabolic, hyperbolic or other shape; surface resolution. Furthercharacteristics are surface response characteristics such as color,texture, depth pattern.

There are a number known methods that the AGUI can use for mapping anon-planar surface. These include stereoscopic optical systems,plenoptic multi-lens array systems and optical systems where either astructured light pattern is projected onto a surface or a scanningsystem sequentially creates a structured light pattern by scanning apoint of light onto the surface and the surface is imaged by an opticalsystem displaced from the projecting system.

In the case of the structured light method, because the optical axes ofthe projection system and the imaging system are not coincident thedifference between the expected and observed positions of points in thepattern can be used in a triangulation method to calculate depthinformation and so, in conjunction with the 2-dimensional locationinformation collected by the imaging sensor, a 3-dimensional map of thesurface can be determined.

One state-of-the-art method for the AGUI system to use to map anon-planar surface is a ‘time-of-flight’ camera system. Such a camerasystem determines the location of object points in a plane perpendicularto the line-of-sight of the camera using the captured image just as isdone with normal camera images. The distance to individual object pointsalong the line-of-sight is found by illuminating the object with anextremely short duration pulse of light. This can be achieved using LEDor lasers that can be turned on and off with times on the order ofpicoseconds. Each pixel sensing element of the ‘time-of-flight’ camerais connected to gated receiving elements and the ‘gates’ are opened incoordination with the pulse of illumination. It takes light a finitelength of time to travel to an object point, be back scattered, andreturn to the camera sensor. The farther the object point is from thecamera, the longer the delay and since light can only be received duringthe time the gate is open, as the time of delay changes, the amount oflight collected changes because some of the light reaches the cameraafter the collection ‘window’ has closed. This effect allows the camerasystem to calculate the distance between the camera and each objectpoint.

To achieve a sufficient signal when using such short pulses of light avery bright illumination source is needed, hence the use of lasers orvery bright LEDs. In addition, the wavelength of light used is typicallyin the near infrared, as very bright sources are available in thiswavelength range and the light is almost outside the visual range sothat a bright source is not uncomfortable or distracting to the humanuser.

For the purpose of mapping the shape of a non-planar display surface, akey factor is not the absolute distance to each object point but thedifference in distance between points. These values can be found bysetting the timing of the receiving ‘windows’ so that some object pointis found to have a zero distance. Then the distance found with thisarrangement for all the other points is the difference in distance fromthe reference point (the point with zero distance).

Once a non-planar surface is characterized by having three coordinatevalues found for each object point, the surface is mapped. In terms ofdata that can be used by a computer system, the information can bethought of as existing as a 2-dimensional image with an intensity valueat each location in the image. The locations in the 2-dimensional arraycan be the object points where rays of light from locations in the imagesensor intersect the object. This gives the 2-dimensional location ofeach object point in a plane perpendicular to the camera line-of-sight.The intensity value then gives the distance difference value, or thirddimension, of the object point from some reference distance point in theobject.

The current information on the 2-dimensional location of object pointsdefines for the AGUI system the available surface to create a displayand allows it to best create a display with this available surface. Thedifferential distance information, the third dimension, is used by theAGUI system to compare current object distances with prior objectdistances and so, in conjunction with lateral movement information, totrack motion between points in the non-planar surface. For instance, theAGUI system uses this information to detect control movements.

In some embodiments, special use of ‘time-of-flight’ camera systems maybe used with embedded source fabrics. For non-planar display surfacessuch as garments with addressable OLEDs embedded in them anotherpossibility exists for surface mapping using a ‘time-of-flight’ camerasystem. It is possible to embed in such a garment fabric not only thevisible light OLEDs that create the visible display but also an array ofdiode lasers or near infrared LEDs that can be pulsed with very shortpulse durations. Then instead of having the ‘time-of-flight’ camerasystem provide the special illumination, the garment itself provides itsown special illumination. Now, since there is no need to have backscatter and the loss of source light this entails, the sources do nothave to be as bright, or if they are very bright, the signal to noiseratio of the system is markedly enhanced.

If such a display system were detected by the AGUI system that also hadaccess to a ‘time-of-flight’ camera system, the AGUI system wouldcollect from the display system information on the pulse length of thespecial sources and time between pulses. The AGUI system would theninstruct the ‘time-of-flight’ camera system to set its gating properlyto coordinate with the sources in the display system. Then, with thespecial sources in the display system pulsing, the AGUI system wouldmove the timing of the gating until at least one of the sources wasfound to have a zero distance. At this point full mapping would proceedas described above for a normal ‘time-of-flight’ camera system.

Some devices with non-planar display surfaces may have the ability toself-map their display surface. For instance, a garment with embeddedvisible light emitting elements, such as OLEDs, could also have sensorsembedded that could sense the distance to nearest neighbor locationsensors. This differential distance information is then used toconstruct a 3-dimensional net, thus mapping areas of the surface. Eachlocation sensor would have a neighborhood of visible light emittingelements, whose position would be calculated with respect to the3-dimensional net using interpolation methods. The device with this typeof self-mapping capability then will have continuously updated3-dimensional location information on each display element. When theAGUI system establishes communication with the display device and sensesit is a self-mapping device, the AGUI system will command the displaydevice to transmit the 3-dimensional pixel information. Then the AGUIsystem will have information it needs to find the shape of the availabledisplay surface and to command an appropriate display to be created. Inaddition to the 3-dimensional surface representation, the AGUI systemwill need to determine the orientation of this surface to the viewer. Ifthe AGUI system has the capability to capture, it captures images of thedisplay surface so it will have a representation of the display surfaceas viewed by the user. The AGUI system will then perform a 3-dimensionalcoordinate transformation of the transmitted surface element locationdata until the transformed data matches the image of the surface. Thisestablishes the orientation of the surface to the viewer. If an imagecapture system is not available, the AGUI system will assume the vieweris located with the best view of the display surface and then proceed tocreate the best possible display.

4.3. Surface Mapping Algorithms

The AGUI system may be operating on a device that does not haveprojection capabilities. However, the device may have other types ofinteraction sensor systems such as those currently found on mobiledevices, desktop and laptop computer systems that employ glide padsensors, touch screen or mouse user input control systems or it may bein communication with devices that have these capabilities. When theAGUI system recognizes that it is operating in such an environment, itwill adapt its interaction and control actions to fully utilize thesecapabilities.

The AGUI system may also find that it can communicate with a displaydevice whose physical shape can change. Such a display device may evenbe able to self-map itself as its shape changes. An example of a displaydevice that can change the shape of the display surface and can self-mapthe shape changes is a garment embedded with OLED elements that arecommanded to illuminate to create a desired display. The OLED displaydevice has location sensors embedded in it that communicate with oneanother thus giving real-time information on the 3-dimensional positionof each location sensor with respect to its neighbors. The displaydevice also has information on the location of each OLED with respect tothe locations sensors. An array of points whose locations with respectto one another are known constitute a mathematical 3-dimensionalsurface. Thus the display surface is self-mapped in 3 dimensions. Inaddition, since the location of the OLED display elements are known withrespect to the location sensors, the OLED display itself is self-mapped.For the AGUI system to make such a surface useful to a user, thelocation of the user with respect to at least three of the locationsensors must also be known. Then the location and orientation of thedisplay surface with respect to the user is known. With this addedinformation on the position of the user to the display surface, the AGUIsystem can command the display surface to create a display that providesan optimal image when viewed by the user of the AGUI system.

The AGUI may alternatively find that it can communicate with a displaydevice whose physical shape can change but does not have self-mappingcapability. In this case, for the AGUI system to use this displaysurface the AGUI system needs to have the capability to map the displaysurface in real-time. Real-time mapping is most conveniently done byequipping the AGUI system with the ability to capture images and thenusing this ability to capture images of the display surface. Then, incommunication with the shape changing display surface control, the AGUIsystem first commands a display to be created and then images thecreated display. Comparison of the commanded display with the imageddisplay allows the AGUI system to map the available display surface andto not only most efficiently use this available surface but also tocontinuously update its mapping to allow for shape changes that mayoccur.

In an AGUI system, several classes of displays are possible.Self-mapping can be limited to a 3-dimensional correlation of thedisplay elements to the surface or can also include an addition 3degrees of rotational freedom with respect to a line-of-sight betweenthe viewer of the display and the display itself. If self-mapping islimited to a 3-dimensional correlation between the surface and itsdisplay elements, the AGUI system will decide to create a display thatbest uses the available display surface and will assume that the viewerfaces the surface with the best orientation for viewing. If the displaysurface is the display screen of a desktop or laptop computer, the3-dimensional mapping would give the location of the display pixels withrespect to a 2-dimensional plane. However, in most cases the orientationof this surface to the viewer will not be supplied by the display deviceto the AGUI system. If the display is a garment with OLEDs embedded init and had location sensors so that the position of the OLEDs withrespect to a garment coordinate system were to be known, this wouldprovide a 3-dimensional map of the display surface elements. However,the orientation of the garment to the viewer might not be informationthat the garment device could provide to the AGUI system.

If the AGUI can receive some mapping information from the display deviceand in addition can view the display being detected, the AGUI will makethe decision to use the mapping information in addition to imageinformation to establish not only surface display element mapping withrespect to the surface itself but also will map the surface with respectto the viewer's line-of-sight before proceeding to the next step. Ifthis cannot be done, the decision to take the alternate path asindicated it made.

It may well be that the display device has no self-mapping capabilities.In that case if the AGUI can view the display the decision is made tomap the surface itself and then proceed to choose the scenario bestsuited to the display and its orientation. If the AGUI system is unableto do this, the decision to choose a scenario in which an assumption ismade that the viewer is facing the surface is made. The surface isoriented so that it display surface can be best seen by the viewer. Ineither case, once the scenario has been chosen the AGUI process willproceed to its control and update portion.

Another possibility is that although the AGUI system cannot map thesurface including its orientation to the viewer it may be able toestablish the location of the display device with respect to itself. Forinstance, both the AGUI system and display could have GPS locationcapabilities so a line-of-sight between the viewer and the display canbe established using the information. If in addition the display devicehas the ability to measure its orientation and transmit this informationto the AGUI system, this is sufficient information for the AGUI systemto select the best scenario to use the display device. If this type ofinformation is not available, the AGUI will decide to choose a scenarioin which an assumption is made that the viewer is facing the surface andthe surface is oriented so that its display surface can be best seen bythe viewer. In either case, once the scenario has been chosen the AGUIprocess will proceed to its control and update portion.

4.4. Surface Types (Hand, Clothing, Automotive, . . . )

An example embodiment is the display of content onto an array of OLED'sembedded into clothing. Content sent to the OLED array may be acombination of text and graphics. The OLED's in the array are aware ofthe identity and separation of adjacent OLED's. This separation data isthe basis of the determination of a geometric transformation thatremoves the distortion that would be caused by the geometricorganization of the OLED's not being rectilinear.

AGUI systems may include adaptive clothing with the ability to adapt toa flexible multi-sensor display surface and adapt content on the mappedbody or clothing in relation to optimum viewing by one or more otherpeople. In these embodiments, customized content may be displayed ondifferent sections of the clothing such as a phone interface on a jacketsleeve or mapping and assigning a specific advertisement or graphiccontent on the back of a jacket based the relational position of one ormore viewers.

Some embodiments of AGUI systems comprise UAVs (Unmanned AerialVehicles). These include the ability for a moving projection system tocontinuously adapt the angle of projection or continuously adapt graphicand interactive content onto one or more mapped surfaces. This is basedon the spatial position of a moving aerial projection and/or activedisplay system in relation to one or more mapped objects or surfaces orthe relation of one or more viewing parties to the aerial displaysystem.

AGUI systems may include automotive and other vehicular applications.These include single or multi-directional location, orientation andevent specific mapping, imaging, projection or active displays. Thesesystems may interface or network with multimedia display hubs and otherAGUI systems.

4.5. Use of Multiple Surfaces

The device also has the capability to drive more than one surface. Whilethe built-in projector is coupled to the physical surface that iscurrently in its light path, the device can support multiple wired orwireless displays.

In the simplest configuration, the device replicates its output to eachsurface that it is currently connected to. The first level ofenhancement is for the device to reorganize the output optimally foreach of its different surfaces. The device may also recognize that it isconnected to a set of displays of compatible type, and that its outputcontent can be spread from device to device, providing a smaller numberof larger effective surfaces. The device may also reorganize contentbased on the strengths of each connected surface, and offer modifiedcontent uniquely tailored to each. For example, a TV might show a video,while a passive horizontal surface might show a virtual keyboard fordata entry. When the AGUI device recognizes closely positioned oroverlapping display surfaces such as a TV or other active displaymounted on a wall the AGUI display system may communicate with theactive display while projecting content around the active display togenerate a single coordinated image, media content or interactivegraphic user interface across the active and passive display surfaces orthe AGUI display system may assign distinct content to the activedisplay and passive display surfaces or a combination of coordinated anddistinct content across active and passive display surfaces.

When an AGUI device connects to one or more additional AGUI devices,further possibilities become available. Just as a single AGUI canproject the same content onto multiple surfaces, several AGUI's cancooperate and show the same content. Alternatively, as in the moreadvanced configurations of a single AGUI, a group of connected AGUI'smay collaborate to split content, and split functionality. Each AGUIdevice has the capability to operate autonomously or in activitysynchronized with other AGUI devices.

The adaptive display system may also select to assign multipleinterfaces to a single display surface such as a table or wall based onthe position of each person in relation to the display surface so thatmultiple people can be interfacing on a single display surface and userspecific content can be assigned to zones on one or more displaysurfaces.

4.6. Managing Multiple Users

The AGUI system may identify one or more users in close proximity to oneanother, remotely connected or independent of one another. The AGUIsystem may manage a multitude of users and assign independent accounts,operating systems and/or adaptive graphic user interfaces (AGUI's) toeach user, to a group of users or to an organization or company. TheAGUI system may identify a user using one or more methods ofidentification such as imaging and/or mapping the hand, face, eyes,body, heart rate, using voice recognition or other biometric identifiersand/or through password, voice command, touch, gesture or other storeduser input. The AGUI system may also identify a user by a user'sconnected device, clothing or other handheld, wearable, mobile,embedded, implanted, vehicular or other wired or wireless networkeddevice and/or sensor that may also incorporate its own identificationsystems, network identification and/or methods of identifying theirassigned user or users. The AGUI system may also identify the size,location, spatial and relational position and orientation of one or moreusers to one another and/or to one or more active or passive displaysurfaces with or without verifying the identify of one or all of theusers and make autonomous decisions as to how to optimize content forone or all of the users

Each user may have a customized AGUI interface that is adapted to theage, height and other physical characteristics, color and layoutpreferences and/or unique tasks, activities and interest of each user.

When the AGUI system identifies two or more users and identifies theusers environment the AGUI system may adapt a graphic user interface orother multimedia content to each user based on the location, spatialposition, orientation and point of focus of each user in relation toeach other and/or in relation to one or more handheld, wearable or otherdevices, objects and/or surrounding display surfaces. When a user movesfrom one area to another or when a user changes their position,orientation or point of focus in relation to one or more displaysurfaces, the AGUI system may follow each users movements, spatialposition, orientation and point of focus moving the AGUI display fromone display surface to another and/or one AGUI device to another and/oradapt the displayed content based on the type of display surface inwhich each user is interfacing. For example if two users are in the sameroom such as a kitchen and one user is facing a refrigerator and anotheruser is in front of a stove the AGUI system may display content specificto each user and specific to the device, appliance, object or otherdisplay surface in which each user is facing and/or interfacing. Eachdisplay surface such as a stove top or refrigerator door may have aunique AGUI layout and content customized to the size, shape,functionality and/or contents of each device, appliance, object or otherdisplay surface. The AGUI system may adapt the displayed content inreal-time based on each user preferences and the display surface. Forexample one user may drink milk and eat cheese and may have their ownfood items in the refrigerator while another user may be lactoseintolerant and have only non-dairy items in the refrigerator. The AGUIsystem may identify each user when they are standing in front of therefrigerator and display content and information specific to each userspreferences and food items contained within the refrigerator specific toeach user and identify which items are assigned to each user and whichitems are group items. The AGUI system may also map and displayinformation specific to food or other items contained within or specificto the refrigerator or other device, appliance, object or other displaysurface. For example the AGUI system may image, map and/or wirelesslyidentify the food items or other contents of an appliance such as arefrigerator using one or methods of item identification such as objector logo/image recognition, by imaging a barcode, QR code, or othercoding system, by wirelessly connecting with an RFID chip or othersensor assigned to each item or by another method of device, object oritem identification. The AGUI system may image and/or map the spatialposition, contents and status of items placed inside a refrigerator,oven or other device, object, appliance, vehicle or space such ascloset, drawer, cupboard, cabinet, trunk or other storage area andidentify how long an item has been in the space, the temperature of anitem, how long an item has been cooking, how much of an items contentsare left in the container, or other information on the status of adevice, appliance, object, item, person, vehicle or other space ordisplay surface and enable one or more users to interface with and/ortake actions regarding such items such as remotely ordering a new item,turning on or off or changing the temperature of refrigerator, oven orother device or system or beginning, ending or controlling a task suchas brewing coffee, or altering the temperature of a stove top based onthe status of a task in progress. These tasks may be controlled directlyby a user or may be programmed or assigned to the AGUI system to makeautonomously based on a set of defined parameters such as the length oftime an item has been stored, the amount of an item left in itscontainer, the items temperature or any other variable assigned to atask. The AGUI system may then order a new item directly or may alertone or more users as to the status of an item or task in progress eitherthrough a visual or audio alert or via a text, email, phone or othermessage, communication or alert.

The AGUI system may image, map, monitor and/or communicate with itemsdirectly or may network with connected devices, objects, appliances,vehicles or other systems and surfaces such as a computer, phone,television, connected refrigerator, oven, coffee maker or other device,object or appliance or a connected space such as a home, office,building, store, restaurant, gym, theatre or other space or a car, bus,train, plain, boat or other vehicle which may have its own sensors andmethods for identifying and monitoring the items, persons and or othercontents within or outside of the connected system,

The AGUI system may support a single group account in which multipleusers all have the same or different levels of access to a single AGUIplatform with standard or unique assigned AGUI layouts for each user,device, appliance, object or other display surface in a house, office,vehicle or other assigned area or across multiple devices, appliances,objects, vehicles, or other assigned areas or spaces. In this scenariodifferent users may be assigned different levels of access to the coreplatform and/or to the interfaces, controls, programs and/orapplications assigned to each device, appliance, object, area or space.An example would be a household with one or more children of differentages where a younger child may have no access or control over theinterface assigned to a specified device such as an oven and may beassigned limited access or controls to the interface assigned to anotherdevice, object, appliance or other display surface such as arefrigerator wherein the child may only access and interface withcertain sections or tasks of the refrigerator such as the waterdispenser or a vegetable or snack drawer or may only open therefrigerator at specific times set by an authorized user. While ateenager may have unlimited access to the interfaces of all or mostdevices, appliances but may have limited access to the family vehicle orvehicles with access limited to locking or unlocking the vehicle oradjusting the temperature but an authorized user may needed to remotelystart the engine or unlock the steering wheel enabling the teenager todrive. The AGUI system may also be programmed to only allow a specificuser to interface with a device, appliance, object, vehicle or othersystem when an authorized user is present.

The AGUI system may serve as the operating, interfacing and controlsystem for a connected device or may simply operate as an adaptivedisplay and interfacing layer and system for a device with all deviceprogramming, applications, controls managed by the connected device,appliance, object, vehicle or other connected systems own onboardoperating system.

The AGUI system may assign different levels of authority to each userwith regards to editing controls and interfacing access andfunctionality.

In this scenario the AGUI system operates as each users independentmobile adaptive operating system in which each AGUI device may map andidentify each user as they move from one display surface to anotherand/or from one AGUI device to another and continuously adapt contentindependently based on each users distinct account and displaypreferences and based on each users distinct account setup andapplications.

5. CONFIGURATION 5.1. Startup

When power is applied, the system enters a new environment, or thesystem is reset, one of the first actions of the AGUI system is tointerrogate the environment where it is located including the space thatcan be viewed by its image capture system and identify any displays orsurfaces that could be used as a display surface. The AGUI system mayalso identify one or more persons, users and/or viewers in the proximityof one or more of the AGUI devices, vehicles, persons or other objectsor display surfaces. The user may be identified using the AGUI devicesonboard sensors which may include a light and/or optical imaging system,or other biometric identification system and may also include a motion,orientation, location or other sensor to identify the location, activityand position of the device and/or the user. In some embodiments of thepresent technology, this interrogation may be done by projecting ameasuring grid or pattern and seeing if the image capture system recordsan image of this pattern. An image of any portion of the projectedpattern is sufficient to inform the AGUI system that a surface exits. Itis also possible that an image is captured that contains features thathave not been projected but that show that there is a surface in thefield of view of the AGUI system. If, in addition, a communication hasbeen made with a potential display device and the location of thisdevice, person, vehicle or other object or surface is found to be withinthe field of view of the AGUI system, it may be that the availablesurface for display is the display surface of the external device. AGUIsystems, including those that are fixed in place, may store a list ofdisplays and surfaces and attempt to confirm their presence andavailability for use when performing dynamic surface detection.

5.2. Setup and Run Time Configuration

The AGUI system is aware of its surroundings and must be able to detectand configure surfaces and displays on which to output information andcontrols. The system divides displays into two types. The first is thetraditional active or electronic display. Examples include traditionalcomputer monitors, TVs, projectors, and any other device that can bequeried and return data describing its characteristics such as physicalinterface, communication protocol, resolution, and color depth. Theother type is a passive display or surface. A passive surface has noelectrical or electronic connection to the AGUI. Typically, the AGUIdisplays onto a passive surface via its internal projector or anexternal projector. It is associated with input sensors, such as amapping system, that allow for the detection of the surface, reportingrelevant data required to use the surface as a display. The presence andparameters of these passive display surfaces must be detected by sensorsin the AGUI system. Examples of passive displays are hand, arm, table,wall, vehicle, building. All displays are detected, identified, and havetheir properties determined and monitored for updates and changes.

At setup time, the AGUI may prompt the user to provide parameter rangesthat input data is subject to for identifying and selecting surfaces.For example, a horizontal surface need not be absolutely horizontal, buta margin may be established. Vertical surface identification owns asimilar set of parameters. A user may also implicitly set up parameterranges by adjusting existing surfaces set up automatically by thesystem. By overriding default settings, the system can take the userupdate settings as new parameters.

When initializing and setting up the AGUI system user interaction may berequired. Pre-operation setup of the device includes interaction betweenthe user and the AGUI hardware to collect or verify a variety of data.This setup data can include specifying or verifying the list ofavailable surfaces, the list of available ambient sensors, or the listof available gestural interfaces. For each surface, the criteria forselection of that surface as an output device or criteria forrecognition of geometric or response perturbation are determined.

The mapping and characterization of the surface can typically be done byanalyzing a small number of image frames. Once it is done, the AGUI willuse the characterization of the available surface to set the appropriatescenario for the AGUI display. For an AGUI system, the scenario includesthe way in which the display is created so that it is best adapted tothe available surface. For instance, if a human hand scenario is usedthe images fall on the fingers and palm instead of between fingers andcontrol features fit on the fingers and palm with sizes that can bereadily seen by a human viewer and can be manipulated by that person. Inthe case of a hand scenario a projected keyboard would not beappropriate as a control feature. However, if the available surface werea tabletop, a projected interaction keyboard would be appropriate. Ifthe available surface were part of a device that could be commanded bythe AGUI system to create a display yet another scenario would be usedby the AGUI system.

The executing application is aware of the system's input and outputcapabilities and reworks the output content based on several criteriasuch as setup preferences, input data, the currently selected outputsurfaces. Input data can define output surfaces, complete outputcontent, and redefine the executing application.

Setup parameters and run time selections provide capability for drivingmore than one surface. Surfaces may display the same content, with eachsurface receiving modified content appropriately for that surface.Content may be divided between surfaces, so that content is spreadacross a set of surfaces. Content may be organized differently fordifferent selected surfaces, making best use of each surface'scapabilities.

5.3. Choosing Display Surfaces

The AGUI has built in capability to identify a surface type based onprocessed input data. Sensors can determine a number of parameters usinga number of means including light imaging systems, light scanningsystems, rangefinders, depth mappers, optical imaging systems, camerasystems, plenoptic capture systems, networking and sensor mapping bodyarrays, real time streams of planar or non-planar geometric perturbationdata, real time ambient condition sensors, OLED specification data, andspecifications of flexible or clothing-based surfaces.

The first step in this process is to receive raw input data from sensorsdirectly or wirelessly connected to the AGUI. The AGUI then organizesthis data so that at the surface identification level it is seen as AGUIspatial data including coordinates, orientation, curvature, and displaydevice response data such as color and texture.

Other attributes also inform the identification process. For example, ifthe available surface is discontinuous but locally continuous it can bedivided into a pattern of zones that resembles a hand and fingers. TheAGUI may initiate a recognition process for that class of display. Atsetup time, the AGUI interacts with the user to determine appropriateranges for number of zones, size and relative positions of zones. At runtime, these parameters are part of the verification process for a validhand/fingers display.

Run time events also inform the identification process. At setup time,the user may have specified that certain incoming events, such asambient light or temperature, would trigger different surfaceidentification parameters sets.

Once setup is complete, the AGUI makes autonomous decisions aboutsurface selection. When there are several candidate surfaces available,the user may override the autonomous selection. The AGUI records theuser's run time overrides. When it has built up a sufficient database ofoverride selections, the AGUI starts to modify the autonomous processbased on this run time learning. Thus, selection becomes a mixture ofwhat the user has set up and what the AGUI has learned through use.

In the simplest situation, a single AGUI displays its content onto asingle surface. In more complex scenarios, an AGUI is able to displayonto multiple surfaces, and optimize both the content itself and theorganization of that content to match the characteristics of each of themultiple surfaces.

The AGUI system implements a process flow that can project displaycontent onto surfaces and image those same surfaces. The AGUI system canalso be embodied in a device that can sense its orientation and movementin space and its orientation to the direction of its projection systemand image capture system using such available technology as motion,location and orientation sensors, cameras, light, optical and acousticdepth mapping and imaging systems. The AGUI system can also be embodiedin a device that can search for, communicate with, and command otherpotential display devices.

The AGUI system supports several different classes of displays andadapts the output for these classes. Different algorithmic methods canbe employed to process and use several different types of displaysurface shape information.

Devices with flat displays are the type of surface found in desktop andlaptop computers, smart phones and tablets. These display surfaces are 2dimensional planes. When the AGUI establishes communication with thistype of external display device it will receive information on the typeof device and then will recognize it as having a planar display surface.It will also receive information as to the size of the display area andthe number and location of the display pixel elements. If the AGUIsystem has image capture capabilities and an image of the device can becaptured, the image of the display will be compared to the known shapeof the display when viewed directly and the distortion of the image willbe used to find the orientation of the display surface to the viewer.However, in most cases, it will be assumed by the AGUI system that theuser will orient himself/herself to the display for a direct view andwill command displays to be generated using this assumption.

Other devices have non-planar display surfaces and cannot provideself-mapping information. When the display surface is not planar theAGUI system needs information on the 3-dimensional shape of the displaysurface. This information will be used to create a display that mostfavorably uses the available surface and to detect changes in thesurface that may be control signals.

When the AGUI has more than one surface available for use as displays,the system will decide which content is to be placed on each availablesurface in several ways. When the AGUI system recognizes more than oneavailable display it will determine for each if the display is an activeor a passive surface. An active display is one such as the display of adesktop, laptop or tablet computer or a self-luminous display of anytype that may be controlled by the AGUI system via communication withthe device. A passive display is a surface such as a wall, a tabletop, ahuman hand, or a household object such as the surface of a refrigeratoror stove. For a passive surface to be an available display surface aprojector system of some type that the AGUI system can control must beavailable to the AGUI system. Having determined the character of eachavailable display surface the AGUI system will, based on predeterminedinstructions, decide on the appropriate display content to place oneach. However, once displays are initially identified there may well becontrol icons on these surfaces, which when activated, will instruct theAGUI system to change the display on one or more of the availablesurfaces. This feature allows the AGUI to use multiple display surfacesin a dynamic fashion.

The AGUI system will continuously monitor the display surfaces it hasinitially found and determine if they are still available. If a surface,for any reason, becomes unavailable, the AGUI system will reassigndisplayed information as necessary to display surfaces still available.For certain surfaces display content is only needed when they areavailable and when such surfaces become non-available, the AGUI will notcreate that specific display content.

If a surface changes its shape but is still available for display, theAGUI system will determine how best to use this surface, taking intoconsideration its use of all other available surfaces, and change thedisplay on the changed surface as appropriate.

If one or more of the display surfaces available to the AGUI system canmove with respect to their surrounding environment and if the AGUIsystem can determine the surrounding environment in some way, the AGUIcan be instructed to change the display on any one of the surfaces basedon its surroundings as the environment changes. For instance, if theAGUI system includes a laptop computer sitting on an outdoor table infront of the user and a self-luminous display on a passing bus,information from the display on the bus for that user will only bepresented when the bus in the vicinity of the table. Then, as the buspasses, its display system will become unavailable and will leave thefirst AGUI system but may join a second AGUI system further down thestreet and create a different display there.

When the AGUI system has multiple available surfaces on which to createdisplays and has the ability to determine where the user is looking andfrom this information to determine the field of view of the user, theAGUI system can be instructed to only create displays on surfaces thatcan be seen by the user. For instance, if the user walked into a kitchenarea and looked toward the refrigerator, the AGUI could then projectinformation detailing the contents of the refrigerator on the door ofthe refrigerator, or if the refrigerator had on its door a self-luminousdisplay panel, the AGUI system could command this information to bedisplayed on the panel. Then, when the user looked away from therefrigerator, the display on the refrigerator door could bediscontinued.

The AGUI may also make use of all available surfaces and map contentonto a portion or the entirety of a space in order to generate a virtualenvironment. The AGUI then dynamically adapts the displayed contentbased on the position, point of focus and field of view of one or moreusers.

The AGUI can be instructed to treat some of the multiple surfacesavailable to it as those on which displays are used only when viewed andothers on which displays are created whether they are viewed or not.

The field of view used by the AGUI system to decide when to create adisplay on certain surfaces can be defined in several ways. If the AGUIsystem includes a head location or eye direction-sensing device of sometype worn by the user, the device can be used by the AGUI system todetermine where the person is looking. This definition of field of viewcan be defined as the personal field of view as it applies to the fieldof view of a particular person. Another definition of field of viewapplies to situations where a moving display surface comes into an areawhere a number of people could view it if they looked at it. Thisdefinition of field of view can be termed area field of view. Forinstance, in the case of a moving vehicle such as a car, bus, truck,train or plane with a projected or self-luminous display on one or moreof its surfaces such as its side, front, rear or roof panels, doors,wheels or windows, the AGUI system will map all available display typesand surfaces on the vehicle and adapt a single graphic display, mediacontent and AGUI to one or more of the display surfaces on the vehicle.The display may cover some or all panels and display surfaces. It mayassign a single graphic display to the entire vehicle, or may assign aninterface and/or distinct graphic and/or multimedia content to a sectionof the vehicle such as the side of the vehicle across all panels,windows, wheels and other display surfaces on the selected side. TheAGUI may also assign a single graphic user interface to a single panel,window, wheel or other display surface, or the AGUI may assign multiplegraphic and/or multimedia displays and interfaces to a section of thevehicle or to one or more panels and display surfaces.

The AGUI system may also dynamically map and display an interactivegraphic user interface across two or more networked vehicles in closeproximity. This is based on the geometrical relationship (spatialposition, location and orientation) of a vehicle and/or vehicle displaysurfaces to one or more other vehicles and vehicle display surfaces. Forexample, if one bus is driving or parked in front of another bus, agraphic user interface may be mapped across both busses. Content may beassigned across the side and/or roof display panels of both buses orvehicles. Certain information may be considered appropriate for displaywhen the bus passes certain locations on its route. A bus could displaytime to various destinations based on the bus stop it was passing. Itcould display advertisements for local businesses in its currentlocation.

Similarly to a conventional computing device controlling multipledisplays, the AGUI is not limited in the number of surfaces it can beconnected to at any given time. It can duplicate the same content oneach surface or divide the content between surfaces. If content isduplicated, it may still be tailored individually for each surface, inseveral ways. Depending on the type of display, surface, size, shape,pixel resolution and other variables, form factors and object, device orsurface functionality the AGUI may choose to send graphic images, video,icons or other interactive graphic content, detailed text or anabbreviated text synopsis. Depending on active surface componentsidentified, the AGUI may choose to send static content or a userinterface with selections. If content is divided, the AGUI will try tofind the best way to use the distinctive characteristics of eachsurface. An identified hand with finger and palm zones suggestsimplementing a selection user interface. A large continuous projectionor display area suggests showing instructional text, graphics, or avideo stream.

5.4 Surface and Display Mapping

Utilizing a light and/or optical imaging and/or depth mapping system anAGUI apparatus may map one or more available surrounding surfaces. Whenpossible the AGUI system will accurately identify and calculate theprecise shape, size and parameters of one or more surrounding passivedisplay surfaces as projection display candidates and then generate adisplay zone based on the mapped parameters for each display candidatein which an AGUI may be assigned. When mapping active display surfacessuch as electroluminescent, backlit, organic, embedded or otherself-illuminated displays such as an LED, LCD, OLED or other activedisplay system, screen, panel or material in which the display output isincorporated in the display surface, the AGUI system may network witheach screen or active display and receive information on the precisedisplay parameters of each active display system. When available, anactive display system may also provide its own spatial position,orientation and/or location data which the AGUI system may incorporateinto an image and/or depth map of its surrounding space to identifyactive display surfaces as independent display candidates and assignedindependent display zones or may be coordinated with a passive displaysurface such as a television screen mounted on a wall grouping with aprojection system to generate a single coordinated active and passivedisplay surface and larger display zone.

When mapping a space using one or a combination of light and/or opticalimaging and/or depth mapping and/or other available method of depthmapping surrounding spaces, surfaces and display candidates and/ornetworking with active display systems, the AGUI System is able to ableto identify and calculate the size, shape, spatial position and displayparameters of one or more passive and/or active display candidates in amapped space.

When image and/or depth mapping is not available the AGUI system may usespatial positioning data from one or more networked active displaysystems to generate a basic positional map of all active displaycandidates in a space and map the spatial position of one or more activedisplays in relation to a primary AGUI apparatus or one of the activedisplay systems may serve as the primary AGUI apparatus for a givenspace.

5.5 Display Zones

Once one or more passive or active display surfaces or spaces have beenmapped, the AGUI system may identify one or more display candidates in amapped space such as a ceiling, floor, walls, tables, chairs or otherfurniture or appliances that may serve as an optimum surface or space toassign an AGUI display.

A single surface, mapped space or other display candidate may beassigned a single display zone or may be assigned multiple display zonesbased on the optimal available display area. For example if a wall ispartially covered by a painting or if a lamp or table is placed in frontof a section of the wall, the AGUI system may assign a primary displayzone for the entire wall with sections blocked out as non-active subdisplay zones and then map, assign, adapt and display content around thenon-active display zones on the wall using a passive surface displaysystem such as a projector. Alternatively the AGUI system may assignsub-zones to a primary zone for each display candidate for example awall may assigned as a single display candidate with a single primaryzone and then sub-zones may be assigned above, below and to the left andright of the picture frame on the wall and different display content maybe assigned to each sub zone.

A single display candidate such as a table may be assigned multiple subzones based on the position of two or more users so that each user maybe assigned their own subzone to display their own customized AGUIdisplay and content. Two or more users may also select to combine theirsubzones to create a group interface zone or may select to generateadditional personalized subzones and assign different applications andcontent to each sub zone.

The AGUI system is able to dynamically map moving surfaces, identifydisplay candidates, generate display zones and subzones, assign anddisplay content to the zones and arrange, modify, map and adapt an AGUIinterface and display content within one or more zones in real-time. Forexample a wearable AGUI apparatus such as a wearable wrist projectionsystem may image and map the users hand and fingers and generate asingle display zone for the entire hand and fingers when the fingers arepressed together and dynamically generate sub zones for each of thefingers when the fingers are apart. The AGUI may assign one AGUIinterface when content is assigned to a single display zone on the handand switch to another AGUI Interface when the user spreads their fingersapart enabling independent functionality, applications, contentassignment and adapted content display to each zone and sub zone inreal-time.

5.6 Modular Computing and Interfacing

Each display zone or subzone may operate as an independent computingplatform or may operate as a display and interfacing module within alarger computing platform in which active display content may beassigned, displayed, controlled and interfaced by one or more assignedusers and naturally moved from one passive or active AGUI displaysurface to another either by the user or by the AGUI system as it mapsthe movements of an assigned user as they move from one AGUI displaysurface or space to another.

6. NETWORKING 6.1. Networking Overview

Various devices can be networked with the AGUI system. Networkinterfaces can conform to a defined AGUI standard or be non-conformantdevices. Non-conformant devices may have a wrapper to appear to be AGUIconformant. The following sections describe a survey of supportednetworking possibilities.

6.2. Networking from the User's Perspective

A single user may interact with a single AGUI. A single user mayinteract with multiple individual AGUIs. A single user may interact witha cooperative network of AGUIs.

Multiple users may share a single AGUI. Multiple users may interact withmultiple individual AGUIs. Multiple users interact with a cooperativenetwork of AGUIs.

6.3. Networking from the AGUI's Perspective

A single AGUI may act autonomously.

Multiple AGUIs may act individually autonomously. Multiple AGUIs mayform a cooperative network.

6.4. Networking from the Display Surface Perspective

A single surface may be owned by a single AGUI. Multiple surfaces may beeach fully owned by a single AGUI.

A single surface may be owned by multiple individual AGUIs. Multiplesurfaces may be share-owned by multiple individual AGUIs.

A single surface may be owned by a cooperative network of AGUIs.Multiple surfaces are owned by a cooperative network of AGUIs.

6.5. Multiple AGUIs in a Network

Less complex embodiments of the present technology include a singleAGUI. The single AGUI device selects a surface, adjusts content for thatsurface, then displays content on that surface. However otherembodiments of the AGUI system are more complex. A single AGUI devicecan control multiple surfaces simultaneously. The AGUI device is often adevice owning a single projector that is able to dispense light. Thisdoes not preclude the possibility that the AGUI recognizes and connectsto other types of surface that do not require illumination. Surfacesthat do not require illumination from the AGUI include local heads updisplays, electrically or wirelessly connected devices such as acomputer, phone, tablet, TV appliance, vehicle, wearable device, orother type of pixel array (e.g. an array embedded in clothing, furnitureor other objects, devices and surfaces).

A system of AGUI's may run a distributed application in which AGUI'sshare both output and input data.

Two or more independent AGUIs, each controlling one or more separatesurfaces, or a group of AGUIs operating as a network controllingmultiple surfaces are other possible configurations. For embodimentsthat incorporate multiple AGUI's in a network, a simple networkconfiguration has one master AGUI device controlling multiple slave AGUIdevices. This is handled similarly to the case of a single AGUIcontrolling multiple surfaces. The master AGUI fully owns its ownprojector and its connected surfaces. The slave AGUIs provide furthersurfaces under its control. These are added to its list of connectedsurfaces. The master AGUI undertakes one further step, to send thecontent to the slave AGUIs.

In a more advanced situation, a network of AGUI's can negotiate to forma peer-to-peer network rather than a master-slave network. Thisconfiguration may involve several activities. One activity is an AGUInegotiating or directing other AGUIs to enable content for display basedon specific capability. Content may be organized to be held on each AGUIas fixed data that may be transmitted from one AGUI to another as neededor read from a common linked database. AGUIs organize entry and exitrequirements from the network. The network is always seen as a temporarystructure that will change dynamically as AGUI devices are turned on oroff, or AGUI devices enter communication range or disappear from range.Individual AGUIs use their local connections and intelligence todetermine its current set of Surfaces.

6.6. Networking Example

FIGS. 62 and 63 show an embodiment of the networking principlesdiscussed in the previous sections. In this scenario, a Master AGUIDevice or Internet or other network based program, platform, website orapplication, labeled “AGUI HUB” (reference 201), is a Master controllerplatform having access to Slave AGUI devices over any kind of network.The network is a vehicle for communication between the Master AGUI HUBand the Slave devices or slave software applications running on slavedevices. The network may be Internet or other wide area network, a localwired network, a local WiFi network, a local peer-to-peer connectionsuch as Bluetooth, or any other communication system that permits datatraffic.

The Master AGUI HUB may own display devices and surfaces or may not ownany display surfaces, but only direct display activity on Slave AGUIs.Such Master directive activity includes sending content to the SlaveAGUIs and working with each Slave device to optimize its displayactivity. In some cases, a Slave may be able to make all necessarydisplay choices without intervention. In other cases, a Slave may itselfhave display choices that it cannot resolve, requiring the Master to beinvolved in choosing appropriate surfaces and/or assisting with detailedcontent choices. All AGUI content and functionality may be storedlocally on a Master AGUI Device or all AGUI content and functionalitymay be stored and controlled remotely on the Master AGUI Hub or acombination of local and remote data storage and functionality.

The Master AGUI HUB may also have direct access to surfaces (reference1308) such as phones, computers, TVs, furniture, appliances, buildings,roads, vehicles and other display surfaces. Additionally, the MasterAGUI Device or HUB may control displays on the user (reference 1310)such as the hands, fingers, arms and other body surfaces as well aswearable devices and systems such as smartwatches, heads-up-displaysglasses, headsets, clothing, prosthetics and implants. In a largercontext (reference 1312), the Master AGUI may be in communication withvehicles, vehicle devices, vehicle surfaces, robotic assemblies or othermobile and/or moving systems. In another scenario (reference 1314), theMaster AGUI may be in communication with Slave devices, vehicles,buildings, stores, businesses, billboards and/or other indoor andoutdoor displays, display systems and/or surfaces

Such an organization of a Master and a set of Slave devices is at alltimes reconfigurable as devices are turned on and off, come into rangefor activity, or exit that range. Furthermore, such an organization mayitself be a node in a larger organization of smaller organizations.

The Master AGUI HUB may also assign content and/or transfer content fromone or more devices, vehicles, buildings, stores, businesses, billboardsor other indoor or outdoor displays, display systems and/or surfaces toone or more other devices, vehicles, buildings, stores, businesses,billboards and/or other indoor or outdoor displays, display systemsand/or surfaces.

In this example embodiment, the AGUI HUB Master recognizes that it is incontrol of several Slave AGUIs (reference 1306). Each of these may betied to one or more passive or active surface (reference 1304) such ashousehold appliances. The Master only makes detailed display choiceswhen a Slave AGUI is unable to make the choice without intervention.

7. MULTI-USER OPERATION

Special further consideration is now given to a situation where the AGUIsystem has within its current ensemble of display surfaces a number ofdisplay devices or has a large display surface, such as a table top,where a multitude of different displays may be active at once, and wherethe AGUI system recognizes that more than one user is able to view andinteract with one or more of the available displays. For the AGUI systemto deal with this special situation it must first be informed thatmultiple users exist. It must then decide how best to allot displaysurfaces to the multiple users. The AGUI system can obtain thisinformation and use it in a number of ways.

7.1 Identifying Users

The AGUI system begins the process of dealing with multiple users byestablishing that there are indeed multiple users. This can be done in anumber of ways.

If an individual has a device associated with their person that cancommunicate with the AGUI system, the AGUI as it continuallyinterrogates its surroundings, will detect this device and when it doesso, it will attempt to establish that person's identity and location.Such devices include cell phones, wrist mounted devices, and tabletcomputers. These are devices that persons carry, have personalidentification as to their owners and communicate with external systems.If an AGUI system can use such information to make an identification,this individual is included as an identified user within the system.Some devices can not only provide identification for a person but canalso provide information on their location. In that case, the AGUIsystem will use this information to track the location of the identifiedindividual as they interact with the AGUI system.

For an AGUI system that can image an extended area, it will use imagerecognition techniques that are well known to find persons in its fieldof view. If the AGUI system is communicating with a powerful enoughcomputer system or network and if it has access to images of identifiedindividuals, it will attempt to identify persons it finds in its fieldof view. Those individuals that can be identified are added to the knownusers within the AGUI system and other unidentified individuals areadded as unknown but located users.

Of those unidentified users some may be later added to the identifiedusers in the following way. The AGUI system will, as will be outlinedbelow, create a display that can be conveniently accessed by each of thevarious current users. The AGUI system will also make available on thedisplay for an unidentified user the ability for that person to informthe AGUI of his/her identity. This will be done with adequate securitymeasures so that unidentified users will not be allowed to inform theAGUI system of a false identity. These secure identification methodsinclude iris recognition, voice recognition, fingerprint recognition,facial recognition, recognition of entered passwords and combinations ofthese methods.

The AGUI may also encounter situations where it is informed that thereare multiple users but neither their locations nor their identities canbe established. For instance, this can occur when the AGUI is made awarethat it is being used in a meeting situation and can image the area ofinterest but has not ability to do image analysis on the scene so as todetect individuals in a reliable way.

7.2 Actions Following Identification of Multiple Users

For each identified user, the AGUI will determine which of its currentdisplays and/or display locations is an appropriate one for thatidentified individual to view and use. It will then decide how best tocustomize one or more display for the use of each individual—taking intoaccount potential situations where a display surface must be sharedbetween one or more of the identified individuals.

For instance, if the available display surface is a tabletop and severalidentified individuals are seated at the table, the AGUI can have theability to create a display surface in front of each individual withcontent specific to that individual. The AGUI system, using theafore-mentioned ability to dynamically change the placement of contentbased on changes in the available display surface, can, for instancechange the placement of display information for an identified individualif a portion of the table top allotted to that individual becomesunavailable for display—say, for instance, if a cup is placed in aportion the projected display for that person.

If the person moves within the area in which the AGUI can createdisplays, it will make available a display surface for that individual,changing the location of the display as the person changes location. Indoing so, the AGUI will avoid conflicts that will occur as one personmoves into the area of a display surface allotted to another user.

7.3 Actions Following Detection of Unidentified Users

In this situation there is not enough information available to the AGUIsystem to allow it to customize displays and/or display areas tospecific individuals and in that case it will treat all persons on acommon basis.

However, it can allot display areas in a way that will make itconvenient for the various observed persons to have a display that eachcan use. In this case, the AGUI will make available to each unidentifieduser the ability to inform the AGUI, in a secure way, of the person'sfull identity. In the case of identified users, the AGUI system willmake available display surfaces that change location as the person moveslocation, avoiding conflicts in allotting display surface area.

7.4 Mixed Identified and Unidentified Users

There can also be a situation where there are multiple users present forthe AGUI system but only some can be specifically identified as to theirperson and location whereas the others are only identified by location.The AGUI system will then allot a display to each person but onlycustomize the display that is appropriate for use by a fully identifiedindividual. All other dynamic aspects of dealing with changes inavailable display areas for each person are handled the same for all bythe AGUI system.

7.5 Identity and Location of Users is Unknown

The AGUI system may not be able to identify the location of multipleusers but may be made aware that there are multiple users in some otherway. This is a commonly occurring situation where the multiple users areassembled at a meeting. Then the AGUI system will allot display space ina way so that as many as possible of the multiple users can view it.

7.6 Multiple Users in a Vehicle

Another example of the AGUI system allotting different display surfacesto multiple users, sometimes individually identified, is the case of theinterior of a vehicle, such as an automobile, where there is a driverand there may be passengers in the front and in the back seats of thevehicle, each identified by their seating placement in the vehicle. TheAGUI system can have access to various built-in sensing devices in thevehicle such as seat occupancy sensors, interior imaging systems andvarious devices carried personally by the occupants of the vehicle. TheAGUI system uses these sensors to identify the presence, location anddesignation of duty or function in the vehicle. For instance, the AGUIsystem will sense the presence of the driver and assign displays createdin various ways to provide the driver with information needed for thetask of a driver. These include heads-up displays and various dashboarddisplays in addition to appropriate audible information. The passengerswill also be identified by the AGUI system by their presence andlocation in the vehicle and will be assigned displays that they canreadily access and interact with in the fashion heretofore described.

7.7 Access to Personal Data and Security Considerations

When the AGUI system identifies and allots display space to multipleindividuals it can also provide to those individuals whose identitieshave been established, via the display allotted to them, access topersonal information and/or data that may reside elsewhere in devices orsystems with which the AGUI system can communicate. These situationsinclude allowing an individual to access personal information and/ordata on a remote system such as an internet ‘cloud’ system on anothercomputer system—local or remote. In this way, the AGUI system, in thecase of multiple users, can allow each individual user to accesspersonal information in a way that is only available on the displayallotted to them.

When the AGUI system provides personal information to a display surfaceand when it is creating or using multiple display surfaces for multipleusers, it employs security measures so that information for a givenindividual is not displayed on the display allotted to anotherindividual. Thus each of the display areas allotted acts as though itwere an individual display device that is not in communication withother individual display devices.

Of course there are situations in which this separation of access is notadhered to. For instance, when the AGUI is facilitating a ‘gaming’situation in which multiple individuals use a common ‘gaming’ platform,the AGUI will allow different users to access and control the displaysthat are common to users involved in the ‘game’.

8. AVOIDING CRITICAL FIELD OF VIEW INTERFERENCE

When the AGUI system creates a display, such as a heads up display, theAGUI system will have the additional responsibility to recognizecritical fields of view in which it would be unsafe to place displayscausing them to interfere with ability of the user to view objects inthat portion of the total field of view. Situations of this type canoccur when the AGUI creates such displays for the pilot of an airplaneor the driver of a car. In these situations, there are field of viewareas in which critical external events are seen and which the user mustrapidly recognize and correctly take control action.

Any display that will interfere with the ability for the user to see andreact to events so occurring must be avoided. So the AGUI will place theheads up or virtual display in areas that will not conflict with userability to see and react to external events. This is best done byequipping the AGUI system with the ability to map and interpret thetotal field of view of the user.

9. ADDITIONAL DESIGN NOTES

In addition, if the AGUI system has the ability to sense the head andeye direction of the user, it can place the display dynamically as theusers turn their heads so that the display can be seen with a minimum ofadditional head or eye movement but so that at no time is a criticalportion of the total field of view interfered with by a display. It isrecognized that the human visual system is quite sensitive to motion inthe periphery of its visual field. So to prevent the user from beingdistracted by rapid changes in a virtual display that is at a givenmoment not in the central portion of the user's visual field, the AGUIwill make changes to the display in a way so any changes are gradualthus preventing undue distraction caused by the changes.

Referring to FIG. 56, embodiments of the present technology of the AGUIdevice, system or method can utilize the following general synopsis ofthe AGUI operation 60. In the exemplary, a general AGUI operation caninclude core processing 62, a dynamic display system and process 64, adynamic graphical user interface (GUI) 66, a change mechanism 68, and asetup process 70.

The core processing 62 of the present technology can include a “Local”computing device including at least one central processing unit (CPU)and software including instructions that can be configured orconfigurable to organize display “Content” in the light of “Context”.Available hardware and recognition of “Users” of the dynamic display 64is taken into account in this process. The Content can be generallydescribed as what is to be displayed and acted on, and the Context canbe generally described as how the material will be displayed. The Localoperation or process of the core processing 62 can be in communicationwith the Context, Content and/or a Futures of the core processing 62,and/or with a “Networked” AGUI system and/or the outside “Word” of thechange mechanism 68, and/or with the “Setup” 70.

Input can be derived from the current operating Context. The Context caninclude “Users” and mapping data. Active “Displays” of the dynamicdisplay system and process 64 may include mapping systems to allow themto optimize usage and to identify User interaction.

Conflicts can be resolved by the AGUI system and/or by a remote systemoperable connected with the AGUI system. Not all potential Content maybe displayed at any given time. This optimization can include selectingfrom available Displays, available Users and available content “Blocks”of the dynamic GUI 66.

Regarding Context of the core processing 62, available Displays such as,but not limited to, projectors, surfaces, electronic devices, etc., canbe recognized and/or characterized. The Context knows or can determinewhich Display devices are currently in use. It knows or can determinehow to drive or control all the available Displays. It acquires mappingdata and identifies User entry such as, but not limited to, gestures,key presses, mouse clicks, etc.

The Context of the core processing 62 can be in communication with theLocal of the core processing 62, the Users and/or Displays of thedynamic display 64, and/or an “Adjusts” system or operation of thedynamic GUI 66.

Regarding Content of the core processing 62, can include the Blocks tobe output, along with possible “Actions” of the dynamic GUI 66 thatcould result from Users interacting with the Content. Content may bestatic or dynamic or a combination thereof. It may include visual, soundand vibration data.

The Content of the core processing 62 can be in communication with theLocal of the core processing 62, and/or the Adjusts, Blocks and/or“Actions” systems or operations of the dynamic GUI 66.

The core processing 62 can include “Futures” including all possiblefuture scenarios that could result from a user Action, or an externalinput from a “Network” AGUI system or from the outside “World” of thechange mechanism 68.

When a Future is selected by the Local processing, it replaces thecurrent Content. A Future may be dynamically defined. The Futures listmay change as a result of Local, Network or World activity.

The Futures of the core processing 62 can be in communication with theNetwork, World and/or Stimuli systems or operations of the changemechanism 62.

The dynamic display 64 of the present technology can include and/ordetermine all available Display devices, along with mapping data thatprovides detailed geometric and response data necessary for cleardisplay. It can further include and/or determine all identified Users.

The dynamic display 64 of the present technology can include theContext, which can be also be included with the core processing 62, an“Adjusts” system or operation that includes mechanisms for adjustingdetails, such as which Displays are currently active, which arecurrently selected, which Users are available to interact with Content,and/or which Users are currently selected to provide input.

Output can be adjusted in real time to accommodate changes in Displaydetails, and devices becoming available and unavailable. Users come andgo into the environment covered by the AGUI system, and outputoptimization changes accordingly.

The Adjusts system or operation of the dynamic display 64 can be incommunication with the Context and/or Contents systems or operations ofthe core processing 62.

Regarding the “Users” system or operation of the dynamic display 64, theAGUI system can work with no identified User. If there is one or moreUser, the Users' locations may be determined from mapping data when thisis available. Display details may be optimized for known Users.

The Users system or operation of the dynamic display 64 can be incommunication with the Context of the core processing 62, the Displaysof the dynamic display 64, and/or the Actions system or operation of thedynamic GUI 66.

Regarding the “Displays” system or operation of the dynamic display 64,the Content of the core processing 62 may span one or more Displays forcontinuous coverage. Mapping data allows Display data to be optimizedfor clarity. This can include geometric transformations and color andintensity controls of the Display.

The displays utilized in the AGUI system can be, but not limited to,passive displays, active displays, watches, cellphones, tablets, TV's,computers, wall surfaces, floor surfaces, ceiling surfaces, multi-paneldisplays, appliances, OLED furniture, projection systems, tiles,multi-panel displays, vehicle surfaces, wheels, windows, garments,headsets, head-up displays, glasses, head mounted systems, body wornsystems, or visors.

The dynamic GUI 66 of the present technology can include the Content,which can be also be included with the core processing 62, the Adjust,which can be also be included with the dynamic display 64, Blocks andActions.

Regarding the Content, this system or operation can include the materialto be outputted. This may be static or dynamic. It may includecombinations of text, video, sound and/or vibration.

Regarding the Blocks system or operation, the Content can be organizedinto component Blocks. The Blocks that are actually output at any timedepends on the state of the current Context. Blocks may have associatedActions.

The Blocks system or operation of the dynamic GUI 66 can be incommunication with the Content of the core processing 62 and/or thedynamic GUI 66, the Displays of the dynamic display 64, and/or theActions system or operation of the dynamic GUI 66.

The Actions system or operation can provide a mechanism for Users tointeract with the AGUI system. Actions may be always available, or maybe associated only with specific Blocks being active. When a Usertriggers an Actions, a Stimulus results, thereby proving an input in theAGUI system.

The Actions system or operation of the dynamic GUI 66 can be incommunication with the Content of the core processing 62 and/or thedynamic GUI 66, the Blocks of the dynamic GUI 66, and/or the Users ofthe dynamic display 64, and/or the Stimuli of the change mechanism 68.

Regarding the Adjust system or operation of the dynamic GUI 66, this canbe a cooperative process that links the operating Context and theContent to be output.

The change mechanism 68 of the present technology can include theStimuli, the Futures which can be included with the core processing 62,the Network and the World.

The Stimuli system or operation can be triggered by local Actions of thedynamic GUI 68. A Stimulus may trigger a Future Content set to beloaded. Additionally, a Stimulus may modify the existing Content setand/or may modify a Future.

The Stimuli system or operation of the change mechanism 68 can be incommunication with the Futures of the core processing 62 and/or thechange mechanism 68, and/or the Actions system or operation of thedynamic GUI 66.

Regarding the Network system or operations of the change mechanism 68, achange request may be driven by another AGUI in a Network incommunication with this local AGUI. The local AGUI system may send arequest to another Networked AGUI.

The Network system or operation of the change mechanism 68 can be incommunication with the Futures of the core processing 62 and/or thechange mechanism 68, and/or the Local system of the core processing 62.

The World system or operation of the change mechanism 68 may acceptexternal change requests from outside the AGUI Network system. Thus, theLocal AGUI system can receive requests and/or control commands fromsystem, devices and/or processes that are not a Networked AGUI system.The AGUI may trigger changes outside the AGUI system.

The World system or operation of the change mechanism 68 can be incommunication with the Futures of the core processing 62 and/or thechange mechanism 68, and/or the Local system of the core processing 62.

The setup 70 of the present technology can include a “Setup” system oroperation. A new AGUI system may be pre-programmed to perform in acertain way with operator preferences. An AGUI system can have Contentloaded via a Setup mechanism 70. An AGUI system may be set up with fixedor dynamic lists of Futures and Displays.

It can be appreciated that the AGUI system can be configured orconfigurable with preloaded settings, parameters, drivers, instructions,tables, databases, etc., or these can be programmed anytime.

It can further be appreciated that a broad general synopsis of anexemplary operation of the AGUI system has been described, which can beutilized in any embodiment of the present technology or later technologyderived from the present technology.

Although the algorithms described above including those with referenceto the foregoing flow charts have been described separately, it shouldbe understood that any two or more of the algorithms disclosed hereincan be combined in any combination. Any of the methods, algorithms,implementations, or procedures described herein can includemachine-readable instructions for execution by: (a) a processor, (b) acontroller, and/or (c) any other suitable processing device. Anyalgorithm, software, or method disclosed herein can be embodied insoftware stored on a non-transitory tangible medium such as, forexample, a flash memory, a CD-ROM, a floppy disk, a hard drive, adigital versatile disk (DVD), or other memory devices, but persons ofordinary skill in the art will readily appreciate that the entirealgorithm and/or parts thereof could alternatively be executed by adevice other than a controller and/or embodied in firmware or dedicatedhardware in a well known manner (e.g., it may be implemented by anapplication specific integrated circuit (ASIC), a programmable logicdevice (PLD), a field programmable logic device (FPLD), discrete logic,etc.). Also, some or all of the machine-readable instructionsrepresented in any flowchart depicted herein can be implemented manuallyas opposed to automatically by a controller, processor, or similarcomputing device or machine. Further, although specific algorithms aredescribed with reference to flowcharts depicted herein, persons ofordinary skill in the art will readily appreciate that many othermethods of implementing the example machine readable instructions mayalternatively be used. For example, the order of execution of the blocksmay be changed, and/or some of the blocks described may be changed,eliminated, or combined.

It should be noted that the algorithms illustrated and discussed hereinas having various modules which perform particular functions andinteract with one another. It should be understood that these modulesare merely segregated based on their function for the sake ofdescription and represent computer hardware and/or executable softwarecode which is stored on a computer-readable medium for execution onappropriate computing hardware. The various functions of the differentmodules and units can be combined or segregated as hardware and/orsoftware stored on a non-transitory computer-readable medium as above asmodules in any manner, and can be used separately or in combination.

FIG. 55 is a diagrammatic representation of an embodiment of a machinein the form of a computer system 1 of an AGUI device, within which a setof instructions for causing the machine to perform any one or more ofthe methodologies discussed herein may be executed. In various exampleembodiments, the machine operates as a standalone device or may beconnected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment or anyother configuration as described herein with reference to any of theembodiments. The machine may any type of device described and/or shownherein in any of the accompany drawings or a robotic constructionmarking device, a base station, a personal computer (PC), a tablet PC, aset-top box (STB), a personal digital assistant (PDA), a cellulartelephone, a portable music player (e.g., a portable hard drive audiodevice such as an Moving Picture Experts Group Audio Layer 3 (MP3)player), a web appliance, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The embodiment of the computer system 1 includes a processor or multipleprocessors 5 (e.g., a central processing unit (CPU), a graphicsprocessing unit (GPU), or both), and a main memory 10 and static memory15, which communicate with each other via a bus 20. The computer system1 may further include a video display 35 (e.g., a liquid crystal display(LCD)). The computer system 1 may also include an alpha-numeric inputdevice(s) 30 (e.g., a keyboard), a cursor control device (e.g., amouse), a voice recognition or biometric verification unit (not shown),a drive unit 37 (also referred to as disk drive unit), a signalgeneration device 40 (e.g., a speaker), and a network interface device45. The computer system 1 may further include a data encryption module(not shown) to encrypt data.

The drive unit 37 includes a computer or machine-readable medium 50 onwhich is stored one or more sets of instructions and data structures(e.g., instructions 55) embodying or utilizing any one or more of themethodologies or functions described herein. The instructions 55 mayalso reside, completely or at least partially, within the main memory 10and/or within the processors 5 during execution thereof by the computersystem 1. The main memory 10 and the processors 5 may also constitutemachine-readable media.

The instructions 55 may further be transmitted or received over anetwork via the network interface device 45 utilizing any one of anumber of well-known transfer protocols (e.g., Hyper Text TransferProtocol (HTTP) etc.). While the machine-readable medium 50 is shown inan example embodiment to be a single medium, the term “computer-readablemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable medium” shall also be taken to include any mediumthat is capable of storing, encoding, or carrying a set of instructionsfor execution by the machine and that causes the machine to perform anyone or more of the methodologies of the present application, or that iscapable of storing, encoding, or carrying data structures utilized by orassociated with such a set of instructions. The term “computer-readablemedium” shall accordingly be taken to include, but not be limited to,solid-state memories, optical and magnetic media, and carrier wavesignals. Such media may also include, without limitation, hard disks,floppy disks, flash memory cards, digital video disks, random accessmemory (RAM), read only memory (ROM), and the like. The exampleembodiments described herein may be implemented in an operatingenvironment comprising software installed on a computer, in hardware, orin a combination of software and hardware.

Not all components of the computer system 1 are required and thusportions of the computer system 1 can be removed if not needed, such asInput/Output (I/O) devices (e.g., input device(s) 30). One skilled inthe art will recognize that the Internet service may be configured toprovide Internet access to one or more computing devices that arecoupled to the Internet service, and that the computing devices mayinclude one or more processors, buses, memory devices, display devices,input/output devices, and the like. Furthermore, those skilled in theart may appreciate that the Internet service may be coupled to one ormore databases, repositories, servers, and the like, which may beutilized in order to implement any of the embodiments of the disclosureas described herein.

As used herein, the term “module” may also refer to any of anapplication-specific integrated circuit (“ASIC”), an electronic circuit,a processor (shared, dedicated, or group) that executes one or moresoftware or firmware programs, a combinational logic circuit, and/orother suitable components that provide the described functionality.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present technology has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the present technology in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the presenttechnology. Exemplary embodiments were chosen and described in order tobest explain the principles of the present technology and its practicalapplication, and to enable others of ordinary skill in the art tounderstand the present technology for various embodiments with variousmodifications as are suited to the particular use contemplated.

Aspects of the present technology are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thepresent technology. It will be understood that each block of theflowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present technology. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions. In otherembodiments, any one or more of the method or process steps or blocksshown in the flow chart may be omitted or re ordered in relation to theother steps or blocks.

In the following description, for purposes of explanation and notlimitation, specific details are set forth, such as particularembodiments, procedures, techniques, etc. in order to provide a thoroughunderstanding of the present invention. However, it will be apparent toone skilled in the art that the present invention may be practiced inother embodiments that depart from these specific details.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” or“according to one embodiment” (or other phrases having similar import)at various places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments. Furthermore, depending on the context ofdiscussion herein, a singular term may include its plural forms and aplural term may include its singular form. Similarly, a hyphenated term(e.g., “on-demand”) may be occasionally interchangeably used with itsnon-hyphenated version (e.g., “on demand”), a capitalized entry (e.g.,“Software”) may be interchangeably used with its non-capitalized version(e.g., “software”), a plural term may be indicated with or without anapostrophe (e.g., PE's or PEs), and an italicized term (e.g., “N+1”) maybe interchangeably used with its non-italicized version (e.g., “N+1”).Such occasional interchangeable uses shall not be consideredinconsistent with each other.

Also, some embodiments may be described in terms of “means for”performing a task or set of tasks. It will be understood that a “meansfor” may be expressed herein in terms of a structure, such as aprocessor, a memory, an I/O device such as a camera, or combinationsthereof. Alternatively, the “means for” may include an algorithm that isdescriptive of a function or method step, while in yet other embodimentsthe “means for” is expressed in terms of a mathematical formula, prose,or as a flow chart or signal diagram.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

If any disclosures are incorporated herein by reference and suchincorporated disclosures conflict in part and/or in whole with thepresent disclosure, then to the extent of conflict, and/or broaderdisclosure, and/or broader definition of terms, the present disclosurecontrols. If such incorporated disclosures conflict in part and/or inwhole with one another, then to the extent of conflict, the later-dateddisclosure controls.

The terminology used herein can imply direct or indirect, full orpartial, temporary or permanent, immediate or delayed, synchronous orasynchronous, action or inaction. For example, when an element isreferred to as being “on,” “connected” or “coupled” to another element,then the element can be directly on, connected or coupled to the otherelement and/or intervening elements may be present, including indirectand/or direct variants. In contrast, when an element is referred to asbeing “directly connected” or “directly coupled” to another element,there are no intervening elements present. The description herein isillustrative and not restrictive. Many variations of the technology willbecome apparent to those of skill in the art upon review of thisdisclosure.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. The descriptions are not intended to limit the scope of theinvention to the particular forms set forth herein. To the contrary, thepresent descriptions are intended to cover such alternatives,modifications, and equivalents as may be included within the spirit andscope of the invention as defined by the appended claims and otherwiseappreciated by one of ordinary skill in the art. Thus, the breadth andscope of a preferred embodiment should not be limited by any of theabove-described exemplary embodiments.

What is claimed is:
 1. An adaptive graphic user interfacing systemcomprising: a sensor component configured or configurable to dynamicallymap an environment in proximity to the sensor component to detect a userand a display candidate; a display output component configured orconfigurable to provide display content on or to the display candidate;and at least one processing unit operably connected or connectable tothe sensor component, the display component and at least one memory, theprocessing unit being configured or configurable to: determine a user'srelational position to the display candidate in real time; identify oneof the display candidate and the user in real time; continuously adaptthe display content based on the user's relational position to thedisplay candidate in real-time as the user moves in relation to thedisplay candidate and/or the display candidate moves in relation to theuser.
 2. The system of claim 1 wherein the display candidate is selectedfrom the group consisting of passive display systems, and active displaysystems.
 3. The system of claim 1 wherein the display content isselected from the group consisting of a graphical user interface,interactive media content, icons, images, video, text, interactive 3Dobjects, environments, buttons, and control affordances.
 4. The systemof claim 1 wherein the system further comprising a microphone andspeaker component in operable communication with the processing unit,and being configured or configurable to receive and/or provide an audiosignal.
 5. The system of claim 1 wherein the processing unit beingfurther configured or configurable to optimize the display content or asecond display content viewable by a second user based on a seconduser's relational position to the display candidate or a second displaycandidate.
 6. An imaging and display system comprising: a sensorsubsystem including at least one sensor configured or configurable tomap an environment in proximity to the system to identify a userposition and a display candidate; a display subsystem including one ofat least one projector configured or configurable to project at leastone display content on the display candidate, and a controllerconfigured to control the display candidate; and at least one processingunit operably connected or connectable to the sensor subsystem, thedisplay subsystem and at least one memory, the at least one processingunit being configured or configurable to: determine one or morecharacteristics of the display candidate; determine motion of an objectin the environment; and configure the display content based on one ofthe characteristics of the display candidate, and the motion of theobject.
 7. The system of claim 6 wherein the system is a wrist mounteddevice with the projector arranged to illuminate at least one of frontand back of a hand wearing the unit, a wrist, forearm or finger of thehand, and the sensor is configured to detect a size or orientation ofthe hand wearing the unit, a wrist, forearm or finger of the hand orentering a field of view of the sensor.
 8. The system of claim 7 whereinthe processor continuously monitors the user hand to detect if a fingeror a second object enters a predetermined area for determining interfaceinputs.
 9. The system of claim 6 wherein the system is eye glasses withthe projector being located to project onto at least one lens of the eyeglasses, and the sensor including at least one eye facing camera and atleast one outward-facing camera.
 10. The system of claim 9 wherein theoutward-facing camera continuously maps the environment, and the eyefacing camera continuously monitors a field of view for determininginterface inputs.
 11. The system of claim 6 wherein the system is avirtual or augmented reality headset.
 12. The system of claim 6 whereinthe system includes a plurality of interconnectable modules, with afirst module including the sensor, and a second module including theprojector.
 13. The system of claim 12 wherein each module includes acommunication port operably connected with a communication port of anadjacent module connected therewith.
 14. A computing device comprising:a sensor for mapping an environment in proximity to the device toidentify a user position and a display candidate; a memory containing anapplication coupled to a processor, the processor executing theapplication to generate display content; the application evaluating thedisplay candidate and the user position and selecting the displaycandidate as a display surface; and a display output, the applicationutilizing characteristics of the display surface and the user positionto modify the display content to produce enhanced display content, thedisplay output displaying an image of the enhanced display content onthe display surface.
 15. The computing device of claim 14, theapplication continuously updating the user position and the displaycandidate to detect a perturbation in the environment, in response tothe perturbation, the application modifying the enhanced display contentto improve the image.
 16. The computing device of claim 14, theapplication continuously updating the user position and the displaycandidate to detect a perturbation in the environment, in response tothe perturbation, the application detecting a control action.
 17. Thecomputing device of claim 14, the application continuously updating theuser position and the display candidate to detect a perturbation in theenvironment, in response to the perturbation, the application detectinga second display candidate and selecting the second display candidate asthe display surface.
 18. The computing device of claim 14, wherein thedisplay surface is a passive surface and the display output is aprojector.
 19. The computing device of claim 14, wherein the sensoridentifies a second display candidate, the application selecting thesecond display candidate as a second display surface, wherein modifyingthe display content further comprises dividing the display contentbetween the display surface and the second display surface to producesecond enhanced display content for the second display surface, thedisplay output displaying a second image of the second enhanced displaycontent on the second display surface.
 20. The computing device of claim14, wherein the computing device comprises a plurality of computingsub-devices networked together.